我在dataproc上运行一个spark作业,它从bucket中读取大量文件,并将它们合并到一个大文件中。我使用的是Google API Services Storage 1.29.0的着色功能。到目前为止,它运行良好,合并了大约20-30K个文件。今天我用大约5倍的文件尝试了它,突然我陷入了僵局(在东部我认为我是,因为似乎我所有的执行者都在无限期地等待)。
这是线程转储:
org.conscrypt.native crypto.ssl_read(native method)
Org.CorcRy.NATIVEL.Read(NATIVIL.JAVA:416)
Org.CysCytP.CONSCRYPTFILIGRESUTPROTSCORIP$SSLIPSPATSWOR.RADION(CONSCRYPFTRIGETRESULTSCORT.Suff.java:547)=gt;保持监视器(Java.Lang.Objject @ 1638155334 })
Java.Io.BuffReDePixStudio.Prime(BuffReDeEndoStudio.java:246)
Java.Io.BuffReDePixStudio.Read 286(BuffReDeEndoStudio.java:286)
Java.Io.BuffReDePixStudio.Read(BuffReDePixStudio.java:345)=gt;保持监视器(Java.Io.BuffReDeEndoSturts @ 1513035694 })
Sun.net .www. HTTP.HtpCopy.PARSETHPHPHEADER(HTTPclito.java:735)
Sun.net .www. HTTP.HTTPclito.PARSEHTTP(HTTPclito.java:678)
Sun.net .www.协议。http.httpullOng.GETInPutsStudio0(httpUllClult.java:1587)=gt;保持监视器(Sun.net .www. http.委派httpSurlCu连t @ 995846771 })
Sun.net .www.协议。http.httpull连接。GETInPvestSo流传(httpUllClult.java:1492)=gt;保持监视器(Sun.net .www. http.委派httpSurlCluts@ 995846771 })
Java.NET.HTTPURLCONTION GETRealSCODCODE(HTTURLCONTION.JAVA:480)
Sun.net .www. task.http.httpSurl连接IMP.GETRealSeCCODE(httpSurnCounIdclip.java:347)
阴影。谷歌。API。客户端。http.javaNET.NEthTTePrimsE.& lt;init & gt;(NethTestPrimsS.java:37)
阴影。谷歌。API。客户端。http.javaNET.NETHTTPRECQUES.ExECUT(NETHTTPRESQuest.java:105)
阴影。谷歌。API。客户端。http:httpQuest.Excice(httpQuest.java:981)
阴影。谷歌。API。客户端。GoGoLePAPI。Services。
阴影。谷歌。API。客户端。GoGoLePAPI。Services。
Client。谷歌。API。客户端。GoGoLePAPI。Services。ActudioGooClilient请求。ExeExtudieMedia(抽象GoGoClilient请求。Java:380)
阴影。谷歌。API。Services。存储。$对象$GET。ExtExtudieMedia(存储。Java:6189)
阴影。谷歌。云。存储。SP1.V1.HtpStasaGr.PC.Load(HtpStasaGr.PC.java:584)
阴影。谷歌。云。存储。存储IMPL $ 16。调用(SurrasIMPL。Java:464)
阴影。谷歌。云。存储。存储IMPL $ 16。调用(SurrasIMPL。Java:461)
阴影。谷歌。API。GAX.ReTest.DigTrrTrimeExrtuutor。提交(DigTrrTrimeExtruutor。Java:89)
阴影。谷歌。云。ReCyHelpR.Run(ReCyHelp.java:74)
阴影。谷歌。云。ReCyHelpor。RunHebug重试(ReCyHelp.java:51)
阴影。谷歌。云。存储。
阴影。谷歌。云。存储。博客。GET内容(BLB.java:455)
.MyStaseApp.Read PosikDead定位(SturgAPI).java:71)
…
最终,我不得不取消这项工作,因为什么也没发生。
知道是什么引起的吗?我尝试同时使用athreadlocal<storage>和a singlestorageinstance in my code,i t does not see to make a difference.通过给它加阴影。到目前为止,它运行良好,合并了大约20-30K个文件。今天我用大约5倍的文件尝试了它,突然我陷入了僵局(在东部我认为我是,因为似乎我所有的执行者都在无限期地等待)。
这是线程转储:
org.conscrypt.NativeCrypto.SSL_read(Native Method)
org.conscrypt.NativeSsl.read(NativeSsl.java:416)
org.conscrypt.ConscryptFileDescriptorSocket$SSLInputStream.read(ConscryptFileDescriptorSocket.java:547) => holding Monitor(java.lang.Object@1638155334})
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345) => holding Monitor(java.io.BufferedInputStream@1513035694})
sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) => holding Monitor(sun.net.www.protocol.https.DelegateHttpsURLConnection@995846771})
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) => holding Monitor(sun.net.www.protocol.https.DelegateHttpsURLConnection@995846771})
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
com.shaded.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
com.shaded.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:105)
com.shaded.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
com.shaded.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
com.shaded.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
com.shaded.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380)
com.shaded.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6189)
com.shaded.google.cloud.storage.spi.v1.HttpStorageRpc.load(HttpStorageRpc.java:584)
com.shaded.google.cloud.storage.StorageImpl$16.call(StorageImpl.java:464)
com.shaded.google.cloud.storage.StorageImpl$16.call(StorageImpl.java:461)
com.shaded.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
com.shaded.google.cloud.RetryHelper.run(RetryHelper.java:74)
com.shaded.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
com.shaded.google.cloud.storage.StorageImpl.readAllBytes(StorageImpl.java:461)
com.shaded.google.cloud.storage.Blob.getContent(Blob.java:455)
my.package.with.my.StorageAPI.readFetchedLocation(StorageAPI.java:71)
...
最终,我不得不取消这项工作,因为什么也没发生。
知道是什么引起的吗?我尝试同时使用ThreadLocal<Storage>一个单人间Storage在我的代码中,这似乎没有什么区别。
