63 Star 516 Fork 154

GVPFire Framework / fire

 / 详情

[fire-1141]createRandomIntStream不支持foreachRDDAtLeastOnce

已确认
缺陷
创建于  
2024-03-27 15:58

val dstream = this.fire.createRandomIntStream()
dstream. foreachRDDAtLeastOnce(_ => {})

异常如下:
24/03/27 15:44:38 ERROR JobScheduler: Error running job streaming job 1711524750000 ms.0
java.lang.IllegalArgumentException: DStream必须为最原始的source流,不能经过transformation算子做转换!
at com.zto.fire.spark.ext.core.DStreamExt.$anonfun$foreachRDDAtLeastOnce$1(DStreamExt.scala:135)
at com.zto.fire.spark.ext.core.DStreamExt.$anonfun$foreachRDDAtLeastOnce$1$adapted(DStreamExt.scala:111)
at org.apache.spark.streaming.dstream.ForEachDStream.$anonfun$generateJob$2(ForEachDStream.scala:51)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:429)
at org.apache.spark.streaming.dstream.ForEachDStream.$anonfun$generateJob$1(ForEachDStream.scala:51)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.$anonfun$run$1(JobScheduler.scala:260)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

评论 (0)

read-to 创建了缺陷
read-to 修改了标题
read-to 修改了描述
longyin319 任务状态待确认 修改为已确认
longyin319 负责人设置为longyin319
longyin319 剩余工时设置为4小时
longyin319 预估工时设置为4小时
longyin319 计划截止日期设置为2024-03-29
longyin319 计划开始日期设置为2024-03-28
longyin319 剩余工时4小时 修改为0小时
longyin319 预估工时4小时 修改为0小时
longyin319 优先级设置为不重要
longyin319 添加了
 
enhancement
标签
longyin319 关联分支设置为branch-2.4
展开全部操作日志

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(1)
Scala
1
https://gitee.com/fire-framework/fire.git
git@gitee.com:fire-framework/fire.git
fire-framework
fire
fire

搜索帮助