Update translations before v2024.2 regression testing

This PR updates translations to the latest, making progress on getodk/central#650.
Before submitting this PR, please make sure you have:

[x] run npm run test and npm run lint and confirmed all checks still pass OR confirm CircleCI build passes
[x] verified that any code or assets from external sources are properly credited in comments or that everything is internally sourced

It looks like a number of Chinese translations changed. I think this is because the translator got access to staging and was able to see translations in context.


No termination when phone disconnected

I believe bluealsa-aplay is meant to terminate after the device is disconnected (if streaming has occurred).
This worked for me when I last tried a few months ago, but no longer works. As a result, the process sometimes needs to be manually killed in order to release the audio card.
This is somewhat frustrating as I'm trying to get to a headless setup on my RPi - I'd rather not have to ssh in and kill the process from time to time. (Out of curiosity, does bluealsa-aplay use any significant amount of power if the device isn't connected, or is it virtually idle?)

I believe bluealsa-aplay is meant to terminate after the device is disconnected (if streaming has occurred)

No, it is not. It should work without termination. However, it will terminate if one kills bluealsa server. One can run e.g.: bluealsa-aplay 00:00:00:00:00:00 and Linux powered device should change into the bluetooth speaker (assuming that BlueZ is configured in a way that allows connection without human intervention - trusting device/entering PIN code).

This worked for me when I last tried a few months ago, but no longer works. As a result, the process sometimes needs to be manually killed in order to release the audio card.

Can you post here how you are using bluealsa-aplay?

Out of curiosity, does bluealsa-aplay use any significant amount of power if the device isn't connected, or is it virtually idle?

Power consumption should be almost zero. There is a negligence amount of work done by kernel when bluealsa-aplay is running - governing poll() call :). When I'm incorporating new features into the bluez-alsa my number one priority is performance - sometimes it might cause side-effects (I haven't seen any, though), e.g. I'm trying no to use mutexes for thread synchronization :)

No, it is not.

Ok. Ignore me. I must have got confused with something else.

Can you post here how you are using bluealsa-aplay?

I'm using a RPi 3 (with a HifiBerry DAC+). I was previously ssh-ing into the pi to connect the bluetooth and start bluealsa-aplay each time I wanted to play music. But now I've realised that by running bluealsa-aplay with 'nohup' and trusting the bluetooth device through bluetoothctl I can avoid doing any setup - I just turn bluetooth on on my phone and play.
Thanks for replying, been using as above for the past few days with no issues. Haven't tested power consumption but I'll take your word that it's low. :)

But now I've realised that by running bluealsa-aplay with 'nohup' and trusting the bluetooth device through bluetoothctl I can avoid doing any setup - I just turn bluetooth on on my phone and play.

Yeap, that's the idea behind bluealsa-aplay. I've started it (bluealsa-aplay) as a proof-of-concept player, however due to submitted issues, right now it is a solid (I hope) BT player :).

Haven't tested power consumption but I'll take your word that it's low. :)

To be honest, I don't know how to make it any lower :D without extreme profiling and inlined assembler code blocks (I'm talking about bluealsa-aplay, because in the bluealsa code there is a room for some optimizations, e.g. poll() with timeout :/).

been using as above for the past few days with no issues

So, I'm assuming, that this issue is resolved, therefore I'm closing it.

You can start bluealsa-aplay as system service with systemd:
/etc/systemd/system/bluealsa-aplay.service:
[Unit]
Description=BlueALSA player
After=bluealsa.service

[Service]
Type=simple
User=root
ExecStartPre=/bin/sleep 2
ExecStart=/usr/bin/bluealsa-aplay 00:00:00:00:00:00
Restart=on-failure

[Install]
WantedBy=graphical.target

The sleep was necessary, because the After seems to not really work with the Raspbian supplied bluealsa.service. Maybe someone with better systemd skills can improve this ;-)


22 update dg glossary

Updated terms that are worth recording in glossary.
Closes #22

LGTM, will have to add more when time goes by


Tutorial link is down

Hey, the tutorial link is down. I'm not sure if this is temporary, anyway could you upload it somewhere?
https://scotch.io/@rui/how-to-build-a-cross-platform-desktop-application-with-electron-and-net-core

Hi @nopara73, thanks for reporting the issue. I believe it's online again.
Cheers.

Yes it is.


Invalid proof generation

What could lead to such mistakes and what to do to avoid them?

Apr 29 14:32:06 mid mina[2883]: {"timestamp":"2021-04-29 11:32:05.820297Z","level":"Info","source":{"module":"Snark_worker__Functor","location":"File \"src/lib/snark_worker/functor.ml\", line 201, characters 10-21"},"message":"SNARK work $work_ids received from $address. Starting proof generation","metadata":{"address":"<IP_ADDRESS>:8301","pid":8641,"process":"Snark Worker","work_ids":[786227321,1071283144]}}
Apr 29 14:32:14 mid mina[2883]: 2021-04-29 11:32:14 UTC [Error] Verifier stderr: "bad verify: linearization_check\n"
Apr 29 14:32:15 mid mina[2883]: 2021-04-29 11:32:15 UTC [Error] Error verifying transaction snark from "Local": $error
Apr 29 14:32:15 mid mina[2883]:         error: {
Apr 29 14:32:15 mid mina[2883]:   "commit_id": "a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a",
Apr 29 14:32:15 mid mina[2883]:   "string": "Invalid proof"
Apr 29 14:32:15 mid mina[2883]: }
Apr 29 14:32:15 mid mina[2883]: 2021-04-29 11:32:15 UTC [Error] Error verifying transaction snark from "Local": $error
Apr 29 14:32:15 mid mina[2883]:         error: {
Apr 29 14:32:15 mid mina[2883]:   "commit_id": "a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a",
Apr 29 14:32:15 mid mina[2883]:   "string": "Invalid proof"
Apr 29 14:32:15 mid mina[2883]: }

@orefkov - do you have logs that you could share to help debug this issue?

@imeckler - any idea on what could be happening here?

I think I encountered this during today:
$ grep 'Invalid proof' mina_logs.out | wc -l
13796

The node seemed to stop producing snarks. I took steps:

Stop the node
Remove .mina-config/snark_pool
Start the node

And it seems to be back to normal operation now. Let me know if there is any info I can provide.

Spoke too soon, looks like my node consistently shows this now:
2021-06-09 07:46:23 UTC [Error] Error verifying transaction snark from "Local": $error
error: {
"commit_id": "a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a",
"string": "Invalid proof"
}
2021-06-09 07:46:23 UTC [Error] Error verifying transaction snark from "Local": $error
error: {
"commit_id": "a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a",
"string": "Invalid proof"
}

@deepthiskumar can you please help with this issue?

We now have a new proof system that hopefully will not encounter this issue. Closing, but please re-open if you're seeing it after the 3.0.0 release.


DB build error on step 11

Hi,
may you help me to find a way to work around these possible issues?
1)an issue related to /software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 368
and ValueError: Problem with 'source' feature:
1..74
/organism="Psf dif8rs
2)an issue related to
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/microbeannotator/database/conversion_database_creator.py", line 82, in create_refseq_to_uniprot
'CREATE INDEX refseq_index ON refseq_to_uniprot (refseq_id)')
sqlite3.OperationalError: disk I/O error
both issues raised when running the following statement:
microbeannotator_db_builder -d MicrobeAnnotator_DB -m diamond -t 8 --step 11 --no_aspera
at the end I have only the following tables:
ls -ltr ./MicrobeAnnotator_DB/*table
-rw-r--r-- 1 mariap3636 meta   193408369 Jun  3 18:55 ./MicrobeAnnotator_DB/uniprot_swissprot.table
-rw-r--r-- 1 mariap3636 meta<PHONE_NUMBER>8 Jun  5 00:08 ./MicrobeAnnotator_DB/uniprot_trembl.table
and this is the final output
(BUSTER)mariap3636@skirit:~$ ls -ltr ./MicrobeAnnotator_DB/
total 60148256
drwxr-xr-x 3 mariap3636 meta        4096 Jun  3 18:48 kofam_data
drwxr-xr-x 2 mariap3636 meta        4096 Jun  3 18:53 temp_swissprot_dat_files
-rw-r--r-- 1 mariap3636 meta   193408369 Jun  3 18:55 uniprot_swissprot.table
drwxr-xr-x 2 mariap3636 meta        4096 Jun  4 01:46 temp_trembl_dat_files
-rw-r--r-- 1 mariap3636 meta<PHONE_NUMBER> Jun  5 00:08 uniprot_to_refseq.txt
-rw-r--r-- 1 mariap3636 meta<PHONE_NUMBER>8 Jun  5 00:08 uniprot_trembl.table
-rw------- 1 mariap3636 meta           0 Jun 22 14:25 microbeannotator.db
drwxr-xr-x 2 mariap3636 meta        4096 Jun 22 16:32 protein_db
drwx------ 2 mariap3636 meta      524288 Jun 22 23:31 temp_genbank<- this is full
-rw------- 1 mariap3636 meta<PHONE_NUMBER> Jun 23 19:56 conversion.db
here I copy-paste what seems to be relevant in the error message I received (I have attached the complete error message file)
Traceback (most recent call last):
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/microbeannotator/database/refseq_genbank_parser.py", line 57, in table_creator
for record in SeqIO.parse(uncompressed_genbank, "genbank"):
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/SeqIO/Interfaces.py", line 73, in next
return next(self.records)
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 516, in parse_records
record = self.parse(handle, do_features)
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 499, in parse
if self.feed(handle, consumer, do_features):
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 470, in feed
self._feed_feature_table(consumer, self.parse_features(skip=False))
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 230, in parse_features
features.append(self.parse_feature(feature_key, feature_lines))
File "/software/conda-modules/5.3.1/envs/microbeannotator/lib/python3.7/site-packages/Bio/GenBank/Scanner.py", line 368, in parse_feature
) from None
ValueError: Problem with 'source' feature:
1..74
/organism="Psf dif8rs
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/software/conda-modules/5.3.1/envs/microbeannotator/bin/microbeannotator_db_builder", line 445, in
main()
Thanks a lot for any feedback.
Maria
microbeAnnotatorLastSteps.txt

I think this happens when the genbank files are corrupted because they haven't been downloaded correctly. I made a custom loop that re-downloads individual temporal files whenever an error occurs, and that fixed it for me


Add graceful exit feature

I think it would be helpful to clarify what situations you want to handle. A couple I can think of:

Dump the most recent state when a job crashes

Could do this by try/catch, or an atexit hook. Question is how to handle MPI (see below).

Allow users to externally cancel a job, dumping the current state.

https://github.com/CliMA/ClimaAtmos.jl/pull/2481 proposes using a sentinel file: this requires users to have write access to the directory.
Buildkite will send a SIGTERM, which Julia can sort-of handle, but people have had mixed results. It looks like it is possible to change it in the agent config to be other signals, so we could change it to a SIGINT (what Ctrl-C does), which seems like it is handled better. You can also increase the grace period (time until it sends SIGKILL, which we can't handle).
If using Slurm directly, then scancel has a --signal option to send a specific signal.

Dump the most recent state when a job is about to time out. Schedulers usually have a mechanism to signal that they're about to do this, e.g.

Slurm has a --signal option, which will send a signal to the job that the time is about to expire.
Most cloud preemptible instances will give you a short notice period (e.g. 30 seconds) before the VM is killed. For example, GCP will let you register a shutdown script that will be called.

MPI
There is some complexity when dealing with multiple processes:

How to signal to all processes that they need to exit gracefully?

It looks like srun should forward signals to all processes, but it looks like SIGINT might be special? Would need to look into this.

In case 1, if one process crashes we need to be able to alert all other processes. Unfortunately, the only mechanism MPI provides is MPI_Abort which doesn't let us exit gracefully. Two options I can think of

Call scancel to send the signal to all other jobs (this is Slurm-specific though)
Have another MPI channel (e.g. tag=0) that we can use to send control messages between jobs (I've sort of wanted something like this for logging and other non-compute workloads).

One additional challenge with MPI is that you can end up in a case where a process is waiting on a receive which will never come. This is likely for case 1, but even in cases 2 and 3, there is no guarantee that all processes will receive the notification to cancel jobs at the same time (e.g. if we only check the existence of a sentinel file every time step, then 1 process may be slightly ahead and miss the notification). The is further compounded by the fact that if we want to write to HDF5 in the exit procedure, that needs to be done collectively over MPI.
There are two options I can see here:

We run the neighbor communication and the HDF5 on separate CPU threads. For this to work, we would need to initialize with MPI_THREAD_MULTIPLE (as we could have two separate MPI calls going at the same time.
We switch the MPI.Waitalls to a MPI.Testall (basically a cooperative waitall, similar to https://github.com/JuliaParallel/MPI.jl/pull/766), which would let the Julia task manager switch between different MPI comms.

I think the most promising case might be to

Use an atexit hook to dump the file on a non-zero exit code.
Set up our CI to send SIGINT on cancellation, as that seems the most well-supported, and is capturable as an exception by Julia. This would probably require setting Base.exit_on_sigint(true) so that we can capture it correctly.
Not sure what to do about MPI: I think the easiest option for now would be to make all MPI communication happen on a different thread (basically change our MPI.Waitall(...) to wait(Threads.@spawn(MPI.Waitall(...))).

Ctrl-C does not reliably work (for me, at least). So, I'd strongly prefer not going down that route.
Whether we dump data or not seems orthogonal to me. Users may just want to leave the stepping loop so that they can interactively investigate the time stepper / solution at that time.
Also, I'd prefer a solution that is not tied to buildkite.
Regarding MPI + crashed / partially crashed simulations, I think a combination of try-catch and a separate MPI channel (mentioned in ii of bullet 2) would be ideal.

(e.g. if we only check the existence of a sentinel file every time step, then 1 process may be slightly ahead and miss the notification).

If we cannot assume that all processors have the same view of the file system, can we do something like:

If a processor (let's say X) crashes (e.g., in saturation adjustment), we can use the try-catch and that processor will gracefully exit
Another processor (Y) may not have crashed and is waiting on Y to communicate (e.g., DSS), if we use tasks to perform that and also look for a kill signal, we can be sure that one of the two will work.

All that said, a lot of complications come from MPI + crashed simulations. For non-MPI runs, the solution in #2481 seems to be pretty helpful with little effort.

Ctrl-C does not reliably work (for me, at least). So, I'd strongly prefer not going down that route.

My investigations lead me to two discoveries:

You need to set Base.exit_on_sigint(false) when using a script (this ensures an InterruptException is thrown).
This will prevent it from exiting out of ccalls (e.g. if you're stuck in a MPI.Waitall). Calling it on a different thread (wait(Threads.@spawn(...))) solves the issue.

Also, I'd prefer a solution that is not tied to buildkite.

Agreed. I also don't like tying it to file system behavior either though.

For MPI, if we cannot assume that all processors have the same view of the file system, can we do something like:
* If a processor (let's say X) crashes (e.g., in saturation adjustment), we can use the try-catch and that processor will gracefully exit

* Another processor (Y) may not have crashed and is waiting on Y to communicate (e.g., DSS), if we use tasks to perform that and also look for a kill signal, we can be sure that one of the two will work.

That would work, but then you can't do any collective operations (like write HDF5 files)

One potential wrinkle with my plan: https://github.com/JuliaLang/julia/issues/52771 (polling via MPI.Testall would still work though)

I don't understand under which conditions using a sentinel file will fail for MPI runs. I understand that we cannot assume
the filesystem to be perfectly identical at all time across different processes, but the output directly will likely be shared and synced. There might be a latency, but eventually all the processes will see that there's a termination file. Isn't this the case?

I don't understand under which conditions using a sentinel file will fail for MPI runs.

One MPI process may be ahead of others, and not see that you've changed the graceful exit bool in the file. Processors late to the game do see it, and then the process that missed it will likely end up hanging while waiting for the other processors.

I don't understand under which conditions using a sentinel file will fail for MPI runs.

One MPI process may be ahead of others, and not see that you've changed the graceful exit bool in the file. Processors late to the game do see it, and then the process that missed it will likely end up hanging while waiting for the other processors.

But wouldn't the faster process just exit at the following iteration?
I agree that we cannot (easily) ensure consistency.

I got it.
It would wait at DSS.

I asked on Julia slack, someone pointed me to this:
https://gitlab.gwdg.de/eDLS/InPartS.jl/-/blob/main/ext/InterProcessCommunicationExt.jl?ref_type=heads
which uses this package
https://github.com/emmt/InterProcessCommunication.jl
Unfortunately, it doesn't appear to be registered, but it might be possible to extract the signal handling machinery (https://github.com/emmt/InterProcessCommunication.jl/blob/87786bb966809b9d0248d24c77a7601a3abe2ba9/src/signals.jl) out to its own package?

We added a graceful exit feature, but it has limitations, so I’ll change the title to fix/improve on the limitstions (MPI)


Array and IteratorObject reduce method type error

🔎 Search Terms
"Array IteratorObject reduce"
🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about 5.6.

⏯ Playground Link
https://www.typescriptlang.org/play/?target=99&ts=5.6.3#code/GYVwdgxgLglg9mABAJwKYBMQVQHgCoA0iAygHwAUAsAFCKLABci5MYMsAhgDZPFEQhkaMFCZ4AlIgC8pEgRp1W7XvNqJ2qZGIDaAXUQAfRAEkomjlDjIA8gCMAVqmj5SqyQG8FiAPTf1wZgBBIQ4ATwA6GABnYOQwljNkcUk0KEEkDWRwtExscmAiJShxAG5EL1T09UTsjCxUfMK2YpKaAF8aGlBIWAQAKhQOMABzBo8vUJhULnREAEZWtUnp2YAmRbplmcQAZkWO6hoIBCi4LlRwrjhh8hz68nIOIltJGUQORABqRFsiAAYiHERmNkotjmBTudLtdbnU8o9nq9ZB9vr9EAAidGAoajcjJUpAA
💻 Code
function reduce<T, S>(
  f: (initial: S, current: T) => S,
  init: S,
  iter: T[] | IteratorObject<T>,
) {
  // if (Array.isArray(iter)) return iter.reduce(f, init);
  return iter.reduce(f, init);
}

function* range() {
  yield 1;
  yield 2;
  yield 3;
}

console.log(reduce((a, b) => a + b, 0, range()));
console.log(reduce((a, b) => a + b, "", range()));

🙁 Actual behavior
The type error that should not occur appears at f in the line return iter.reduce(f, init); as follows:
Argument of type '(initial: S, current: T) => S' is not assignable to parameter of type '(previousValue: T, currentValue: T, currentIndex: number) => T'.
  Types of parameters 'initial' and 'previousValue' are incompatible.
    Type 'T' is not assignable to type 'S'.
      'S' could be instantiated with an arbitrary type which could be unrelated to 'T'.

🙂 Expected behavior
Both T[] and IteratorObject<T> have a reduce method, and the interfaces of the two methods are identical. Therefore, the error should not occur. For example, the type error disappears if the commented line is uncommented. All that line does is distinguish whether iter is a T[] or an IteratorObject<T>. In fact, the code after this conditional branch is identical. Therefore, this error should not exist.
Additional information about the issue
No response

Feels like another #60006... unions of callables are not always callable (e.g., overloads and generics)
Personally I'd just work around it by manually widening to whatever callable type you think it should be in the first place, something like
function reduce<T, S>(
  f: (initial: S, current: T) => S,
  init: S,
  iter: T[] | IteratorObject<T>,
) {
  const i: {
    reduce<S>(callbackfn: (previousValue: S, currentValue: T) => S, initialValue: S): S
  } = iter;
  return i.reduce(f, init);
}


Change MockFlags to be more like the real FlagValues.

Handles use cases of FlagValues that were previously unsupported, such
as deep-copying the FlagValues or substituting the inner flag dictionary.
Changes the behavior of the MockFlags.__setattr__ to no longer set the
present attribute (to be more like the real FlagValues.__setattr__). The new
way to set both the value and present attributes is to use
MockFlags[<flag_name>].Parse(<value>).

@cmccoy PTAL

PTAL

One more question, otherwise LGTM.


fix_task_exec_role_arn: Fix problem where it still tries to create aws_iam_role_policy

Change-Id: Ib507b196870340011e8afdfc1f381ae735ba9099
what

The previous PR (https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/pull/63) was incomplete and gives the following error:

Error: "name" must match [\w+=,.@-]

  on .terraform/modules/ecs_alb_service_task/terraform-aws-ecs-alb-service-task-0.32.0/main.tf line 195, in resource "aws_iam_role_policy" "ecs_exec":
 195: resource "aws_iam_role_policy" "ecs_exec" {

The fix ensures that aws_iam_role_policy and aws_iam_policy_document are not created if task_exec_role_arn is passed in.

why

Fix the bug.

/rebuild-readme

/test all


Disable cgo when compiling agent

The agent would fail to run in a Docker scratch container after being
built within the agent-builder container. This appears to have been due
to dynamic linking against clibs which has been removed by disabling cgo
on build.

To be merged after #147

What is the change in size of the binaries if we publish the completely static binaries?

There doesn't appear to be a change in binary or container size.
With cgo:
timber-agent (linux-amd-64) 8.8MB
timberio/agent:0.6.2                9.52MB

Without cgo:
timber-agent (linux-amd-64) 8.8MB
timberio/agent:0.6.2                9.52MB


docker-storage-setup fails but has already gone half way

docker-1.7.1-108.el7.x86_64
Trying to swap between overlay and devicemapper backend...
Added a new disk to the VM, and edited /etc/sysconfig/docker-storage-setup so it contains:
DEVS=/dev/vdc
VG=docker
Ran docker-storage-setup and...
# docker-storage-setup
  WARNING: Device for PV VP6BxI-8NYd-kpBe-Xlr7-LJLO-dmGo-zR2bSx not found or rejected by a filter.
Checking that no-one is using this disk right now ...
OK

Disk /dev/vdc: 416101 cylinders, 16 heads, 63 sectors/track
sfdisk:  /dev/vdc: unrecognized partition table type

Old situation:
sfdisk: No partitions found

New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/vdc1          2048 419430399  419428352  8e  Linux LVM
/dev/vdc2             0         -          0   0  Empty
/dev/vdc3             0         -          0   0  Empty
/dev/vdc4             0         -          0   0  Empty
Warning: partition 1 does not start at a cylinder boundary
Warning: partition 1 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
  Replacing dev /dev/vdc1 pvid VP6BxI8NYdkpBeXlr7LJLOdmGozR2bSx with dev /dev/vdc1 pvid GsLnMJ348U0S4TTr1l24QgrJYo2ZC1A4
  Physical volume "/dev/vdc1" successfully created
  Volume group "docker" successfully created
Storage is already configured with overlay driver. Can't configure it with devicemapper driver. To override, remove /etc/sysconfig/docker-storage and retry.

If it's going to error-out with some warning about existing overlay, why bother doing anything to the hard drives?  I think you should move that check up to the front of the script.

@jeremyeder
This utility currently does two major jobs.

Grow rootfs
Setup storage

Because we can't setup storage does not mean we should not try to grow root (after adding specified disks to volume group). Hence the check is in second part and we still try to grow root first.
I think this behavior is resonable. We instead need to figure out why partitioning disk, adding it to volume group is giving so many errors in your case.


Allow the app to update the Membership level manual override field

We should change this field so the user can get their member status correct without waiting until the overnight thing runs.

The question will be whether the Drupal module will actually go and get the member status like it's supposed to, or if we'll have to wait for the switch to WordPress.

this is done, but again will have to wait and see what Drupal does.

Also worth noting: because this app does not know what the user's current level is, it could possibly downgrade users. This may not happen in this app at all though, because of lack of authentication.

Fixed that issue. Now we only update the temporary override if the 'level' value is greater than that of the user's current value in salesforce


Tessera 0.11 version is not working

Describe the bug
Tessera 0.11 version with Quorum is not working.
To Reproduce
Steps to reproduce the behavior:

Deploy a Quorum network with RAFT or IBFT consensus and TM as tessera 0.11
The pods do not start as tessera:0.11 image is not found

Expected behavior
It seems that tessera 0.11 was not an official release, so all references to tessera 0.11 needs to be updated to 0.10.4 (0.10.6 is the latest release, but we are going to use an older and hopefully more stable release).
Then quorum network should be tested for both RAFT and IBFT.
Official tessera releases https://github.com/jpmorganchase/tessera/releases

The JDK version for both baf-build latest and jenkins tags needs to be updated to 14 for tessera to work. This means all platforms have to be tested with jdk 14

Tested the BAF quorum deployment with tessera 0.10.4, the deployment works with JDK version 14 doesn't work with Java8
The tessera 0.11 image is still available works with Java 8

For tessera 0.10.4 we have tested using JAVA 14, which would be now our JAVA version upgrade for all platforms
Reference story #965


Enable Custom Exporters

Based off how auto instrumentation uses the exporter environment variables, I enabled the distro to use the entry points for additional exporters.

What's the purpose of this feature? Why do we allow users to use other exporters besides azure monitor?

What's the purpose of this feature? Why do we allow users to use other exporters besides azure monitor?

With our emphasis on using env vars for configuring the distro, it seemed to make sense to use the exporter env vars like how they are used in the auto instrumentation flow. Users may want to export to multiple places or output to the console.
Granted, it is possible to add trace and log exporters after the fact. But this approach would be needed for allowing additional periodic metric readers and seems much more user friendly for the tracing and log cases as well.
Since it is not strictly necessary, we can hold off, but is there a deeper reason for the hesitance?


Json parsing failed when i was using the saveAsBigQueryTable

I was trying to load the data to BigQuery using the below  sample code
val conf1 = new SparkConf().setAppName("App").setMaster("local[2]")
val sc = new SparkContext(conf1)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sqlContext.setGcpJsonKeyFile("gcskey.json")
// Set up BigQuery project and bucket
sqlContext.setBigQueryProjectId("proj_name")
sqlContext.setBigQueryGcsBucket("gcsbucket")
// Set up BigQuery dataset location, default is US
sqlContext.setBigQueryDatasetLocation("US")
Usage:
// Load everything from a table
val table = sqlContext.bigQueryTable("bigquery-public-data:samples.shakespeare")
// Load results from a SQL query
// Only legacy SQL dialect is supported for now
val df = sqlContext.bigQuerySelect(
"SELECT word, word_count FROM [bigquery-public-data:samples.shakespeare]")
// Save data to a table
df.saveAsBigQueryTable("my-project:my_dataset.my_table")
While executing the code i got below error when it was trying to execute the last statement
165037 [main] ERROR org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation  - Aborting job.
java.io.IOException: Failed to parse JSON: Unexpected token; Parser terminated before end of string
Can somebody help me to resolve this issue?

Appears the code saves the data as json then writes that into bigquery. Are you able to review the json file for any issues?

not sure where the problem is here, can you post the entire stacktrace?
id seperate the reading vs writing to identify where the issue is

Hi Sam
Please find below the full stack
Warning: Ignoring non-spark config property: SPARK_SQL_AUTH_ADMIN=admin
0    [main] INFO  org.apache.spark.SparkContext  - Running Spark version
1.6.2
242  [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
404  [main] INFO  org.apache.spark.SecurityManager  - Changing view acls
to: adminmana
404  [main] INFO  org.apache.spark.SecurityManager  - Changing modify acls
to: adminmana
405  [main] INFO  org.apache.spark.SecurityManager  - SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(adminmana); users with modify permissions: Set(adminmana)
776  [main] INFO  org.apache.spark.util.Utils  - Successfully started
service 'sparkDriver' on port 33851.
1073 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] INFO
akka.event.slf4j.Slf4jLogger  - Slf4jLogger started
1104 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] INFO
Remoting  - Starting remoting
1224 [sparkDriverActorSystem-akka.actor.default-dispatcher-5] INFO
Remoting  - Remoting started; listening on addresses :[akka.tcp://
sparkDriverActorSystem@<IP_ADDRESS>:33512]
1228 [main] INFO  org.apache.spark.util.Utils  - Successfully started
service 'sparkDriverActorSystem' on port 33512.
1237 [main] INFO  org.apache.spark.SparkEnv  - Registering MapOutputTracker
1250 [main] INFO  org.apache.spark.SparkEnv  - Registering
BlockManagerMaster
1260 [main] INFO  org.apache.spark.storage.DiskBlockManager  - Created
local directory at
/mydata/Mana_2.3/tmp/blockmgr-f8b8270b-ca86-4119-be30-0a9cd3fbc456
1273 [main] INFO  org.apache.spark.storage.MemoryStore  - MemoryStore
started with capacity 4.1 GB
1315 [main] INFO  org.apache.spark.SparkEnv  - Registering
OutputCommitCoordinator
1422 [main] INFO  org.spark-project.jetty.server.Server  -
jetty-8.y.z-SNAPSHOT
1459 [main] INFO  org.spark-project.jetty.server.AbstractConnector  -
Started SelectChannelConnector@<IP_ADDRESS>:8085
1462 [main] INFO  org.apache.spark.util.Utils  - Successfully started
service 'SparkUI' on port 8085.
1464 [main] INFO  org.apache.spark.ui.SparkUI  - Started SparkUI at
http://<IP_ADDRESS>:8085
1482 [main] INFO  org.apache.spark.HttpFileServer  - HTTP File server
directory is
/mydata/Mana_2.3/tmp/spark-7bb14363-a18e-453c-a00c-3518c171a6ee/httpd-a83d59a5-55c0-4004-94e0-5f7409d187c4
1484 [main] INFO  org.apache.spark.HttpServer  - Starting HTTP Server
1490 [main] INFO  org.spark-project.jetty.server.Server  -
jetty-8.y.z-SNAPSHOT
1492 [main] INFO  org.spark-project.jetty.server.AbstractConnector  -
Started SocketConnector@<IP_ADDRESS>:34847
1493 [main] INFO  org.apache.spark.util.Utils  - Successfully started
service 'HTTP file server' on port 34847.
2166 [main] INFO  org.apache.spark.SparkContext  - Added JAR
file:/home/adminmana/bigquery-1.0-SNAPSHOT-jar-with-dependencies.jar at
http://<IP_ADDRESS>:34847/jars/bigquery-1.0-SNAPSHOT-jar-with-dependencies.jar
with timestamp<PHONE_NUMBER>535
2213 [main] INFO  org.apache.spark.executor.Executor  - Starting executor
ID driver on host localhost
2227 [main] INFO  org.apache.spark.util.Utils  - Successfully started
service 'org.apache.spark.network.netty.NettyBlockTransferService' on port
34778.
2227 [main] INFO  org.apache.spark.network.netty.NettyBlockTransferService

Server created on 34778
2228 [main] INFO  org.apache.spark.storage.BlockManagerMaster  - Trying to
register BlockManager
2231 [dispatcher-event-loop-10] INFO
org.apache.spark.storage.BlockManagerMasterEndpoint  - Registering block
manager localhost:34778 with 4.1 GB RAM, BlockManagerId(driver, localhost,

2232 [main] INFO  org.apache.spark.storage.BlockManagerMaster  - Registered
BlockManager
2944 [main] INFO  org.apache.spark.scheduler.EventLoggingListener  -
Logging events to hdfs://
<IP_ADDRESS>:9000/system/spark-history/local-1507090421561.lz4
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml,
hdfs-site.xml
2975 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using
specified project-id 'publicdata' for input
3012 [main] INFO
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase  - GHFS version:
1.6.1-hadoop2
3725 [main] INFO  org.apache.spark.storage.MemoryStore  - Block broadcast_0
stored as values in memory (estimated size 224.6 KB, free 224.6 KB)
3757 [main] INFO  org.apache.spark.storage.MemoryStore  - Block
broadcast_0_piece0 stored as bytes in memory (estimated size 22.2 KB, free
246.9 KB)
3759 [dispatcher-event-loop-12] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_0_piece0 in
memory on localhost:34778 (size: 22.2 KB, free: 4.1 GB)
3761 [main] INFO  org.apache.spark.SparkContext  - Created broadcast 0 from
newAPIHadoopRDD at App.scala:80
3802 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Bigquery connector version 0.10.2-hadoop2
3804 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from default credential.
3805 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from given credential.
3806 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Fetching key
'mapred.bq.gcs.bucket' since 'mapred.bq.temp.gcs.path' isn't set explicitly.
3807 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using working
path: 'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000'
5124 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
5125 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
5125 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'publicdata:samples.shakespeare' to be exported has 164656 rows and 6432064
bytes
5585 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
5585 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'publicdata:samples.shakespeare' to be exported has 164656 rows and 6432064
bytes
5626 [main] INFO  org.apache.spark.SparkContext  - Starting job: take at
App.scala:93
5643 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Registering RDD 1 (map at
App.scala:88)
5644 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 0 (take at App.scala:93)
with 1 output partitions
5645 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 1 (take
at App.scala:93)
5645 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage:
List(ShuffleMapStage 0)
5646 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents:
List(ShuffleMapStage 0)
5652 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ShuffleMapStage 0
(MapPartitionsRDD[1] at map at App.scala:88), which has no missing parents
5680 [dag-scheduler-event-loop] INFO  org.apache.spark.storage.MemoryStore

Block broadcast_1 stored as values in memory (estimated size 3.2 KB, free
250.0 KB)
5683 [dag-scheduler-event-loop] INFO  org.apache.spark.storage.MemoryStore
Block broadcast_1_piece0 stored as bytes in memory (estimated size 2006.0
B, free 252.0 KB)
5684 [dispatcher-event-loop-13] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_1_piece0 in
memory on localhost:34778 (size: 2006.0 B, free: 4.1 GB)
5685 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 1 from broadcast at DAGScheduler.scala:1006
5689 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 2 missing tasks from
ShuffleMapStage 0 (MapPartitionsRDD[1] at map at App.scala:88)
5690 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 0.0 with 2
tasks
5721 [dispatcher-event-loop-14] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 0.0
(TID 0, localhost, partition 0,PROCESS_LOCAL, 2292 bytes)
5723 [dispatcher-event-loop-14] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 1.0 in stage 0.0
(TID 1, localhost, partition 1,PROCESS_LOCAL, 2292 bytes)
5729 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Running task 1.0 in stage 0.0 (TID 1)
5729 [Executor task launch worker-0] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 0.0 (TID 0)
5737 [Executor task launch worker-0] INFO
org.apache.spark.executor.Executor  - Fetching
http://<IP_ADDRESS>:34847/jars/bigquery-1.0-SNAPSHOT-jar-with-dependencies.jar
with timestamp<PHONE_NUMBER>535
5793 [Executor task launch worker-0] INFO  org.apache.spark.util.Utils  -
Fetching
http://<IP_ADDRESS>:34847/jars/bigquery-1.0-SNAPSHOT-jar-with-dependencies.jar
to
/mydata/Mana_2.3/tmp/spark-7bb14363-a18e-453c-a00c-3518c171a6ee/userFiles-450ff49b-5795-4b1d-af43-a07e069cda5d/fetchFileTemp8938094116660258769.tmp
6728 [Executor task launch worker-0] INFO
org.apache.spark.executor.Executor  - Adding
file:/mydata/Mana_2.3/tmp/spark-7bb14363-a18e-453c-a00c-3518c171a6ee/userFiles-450ff49b-5795-4b1d-af43-a07e069cda5d/bigquery-1.0-SNAPSHOT-jar-with-dependencies.jar
to class loader
6755 [Executor task launch worker-1] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-1/data-.json[82328
estimated records]
6755 [Executor task launch worker-0] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-0/data-.json[82328
estimated records]
6764 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-0/data-.json[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040013_0000_m_000000_0 Status:'
6764 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-1/data-.json[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040013_0000_m_000001_0 Status:'
7639 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.json' of size 13348468 to knownFileSet.
7639 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-0/data-000000000000.json'
which has 13348468 bytes. Records read so far: 0
12585 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000001.json' of size 0 to knownFileSet.
12586 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-0/data-000000000001.json'
which has 0 bytes. Records read so far: 164656
12746 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel  - Got 'range
not satisfiable' for reading
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-0/data-000000000001.json
at position 0; assuming empty.
12747 [Executor task launch worker-0] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Found
end-marker file 'data-000000000001.json' with index 1
12897 [Executor task launch worker-0] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 0.0 (TID
0). 2254 bytes result sent to driver
12906 [task-result-getter-0] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 0.0
(TID 0) in 7197 ms on localhost (1/2)
17728 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.json' of size 0 to knownFileSet.
17728 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-1/data-000000000000.json'
which has 0 bytes. Records read so far: 0
17906 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel  - Got 'range
not satisfiable' for reading
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0000/shard-1/data-000000000000.json
at position 0; assuming empty.
17907 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Found
end-marker file 'data-000000000000.json' with index 0
17909 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Finished task 1.0 in stage 0.0 (TID
1). 2309 bytes result sent to driver
17913 [task-result-getter-1] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 1.0 in stage 0.0
(TID 1) in 12190 ms on localhost (2/2)
17914 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ShuffleMapStage 0 (map at
App.scala:88) finished in 12.215 s
17914 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - looking for newly runnable stages
17915 [task-result-getter-1] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 0.0, whose
tasks have all completed, from pool
17915 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - running: Set()
17916 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - waiting: Set(ResultStage 1)
17916 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - failed: Set()
17918 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 1
(ShuffledRDD[2] at reduceByKey at App.scala:89), which has no missing
parents
17924 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_2 stored as values
in memory (estimated size 2.6 KB, free 254.6 KB)
17926 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_2_piece0 stored as
bytes in memory (estimated size 1658.0 B, free 256.2 KB)
17927 [dispatcher-event-loop-19] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_2_piece0 in
memory on localhost:34778 (size: 1658.0 B, free: 4.1 GB)
17927 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 2 from broadcast at DAGScheduler.scala:1006
17928 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 1 missing tasks from
ResultStage 1 (ShuffledRDD[2] at reduceByKey at App.scala:89)
17928 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 1.0 with 1
tasks
17936 [dispatcher-event-loop-20] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 1.0
(TID 2, localhost, partition 0,NODE_LOCAL, 1983 bytes)
17936 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 1.0 (TID 2)
17952 [Executor task launch worker-1] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator  - Getting 1 non-empty
blocks out of 2 blocks
17953 [Executor task launch worker-1] INFO
org.apache.spark.storage.ShuffleBlockFetcherIterator  - Started 0 remote
fetches in 4 ms
18009 [dispatcher-event-loop-24] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_1_piece0 on
localhost:34778 in memory (size: 2006.0 B, free: 4.1 GB)
18082 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 1.0 (TID
2). 4278 bytes result sent to driver
18085 [task-result-getter-2] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 1.0
(TID 2) in 153 ms on localhost (1/1)
18086 [task-result-getter-2] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 1.0, whose
tasks have all completed, from pool
18086 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 1 (take at
App.scala:93) finished in 0.155 s
18092 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 0
finished: take at App.scala:93, took 12.464855 s
18487 [dispatcher-event-loop-28] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_2_piece0 on
localhost:34778 in memory (size: 1658.0 B, free: 4.1 GB)
18490 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 2
18490 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 1
18556 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapreduce.inputformat.class is deprecated. Instead, use
mapreduce.job.inputformat.class
18556 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using
specified project-id 'bigquery-public-data' for input
18559 [main] INFO  org.apache.spark.storage.MemoryStore  - Block
broadcast_3 stored as values in memory (estimated size 230.6 KB, free 477.4
KB)
18569 [main] INFO  org.apache.spark.storage.MemoryStore  - Block
broadcast_3_piece0 stored as bytes in memory (estimated size 22.5 KB, free
499.9 KB)
18570 [dispatcher-event-loop-29] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_3_piece0 in
memory on localhost:34778 (size: 22.5 KB, free: 4.1 GB)
18571 [main] INFO  org.apache.spark.SparkContext  - Created broadcast 3
from newAPIHadoopRDD at package.scala:111
18578 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from default credential.
18580 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from given credential.
18580 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Fetching key
'mapred.bq.gcs.bucket' since 'mapred.bq.temp.gcs.path' isn't set explicitly.
18580 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using working
path: 'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0003'
19704 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
19705 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'bigquery-public-data:samples.shakespeare' to be exported has 164656 rows
and 6432064 bytes
19979 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
19980 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'bigquery-public-data:samples.shakespeare' to be exported has 164656 rows
and 6432064 bytes
19985 [main] INFO  org.apache.spark.SparkContext  - Starting job: first at
package.scala:113
19986 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 1 (first at
package.scala:113) with 1 output partitions
19986 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 2
(first at package.scala:113)
19986 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage: List()
19986 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
19987 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 2
(MapPartitionsRDD[5] at map at package.scala:113), which has no missing
parents
19989 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_4 stored as values
in memory (estimated size 2.8 KB, free 502.7 KB)
19990 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_4_piece0 stored as
bytes in memory (estimated size 1751.0 B, free 504.4 KB)
19992 [dispatcher-event-loop-30] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_4_piece0 in
memory on localhost:34778 (size: 1751.0 B, free: 4.1 GB)
19992 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 4 from broadcast at DAGScheduler.scala:1006
19992 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 1 missing tasks from
ResultStage 2 (MapPartitionsRDD[5] at map at package.scala:113)
19993 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 2.0 with 1
tasks
19994 [dispatcher-event-loop-31] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 2.0
(TID 3, localhost, partition 0,PROCESS_LOCAL, 2303 bytes)
19994 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 2.0 (TID 3)
19996 [Executor task launch worker-1] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0003/shard-0/data-.avro[82328
estimated records]
19996 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0003/shard-0/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040013_0003_m_000000_0 Status:'
20928 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 3964994 to knownFileSet.
20929 [Executor task launch worker-1] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040013_0003/shard-0/data-000000000000.avro'
which has 3964994 bytes. Records read so far: 0
21707 [Executor task launch worker-1] WARN
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Got
non-null delegateReader during close(); possible premature close() call.
21709 [Executor task launch worker-1] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 2.0 (TID
3). 2283 bytes result sent to driver
21710 [task-result-getter-3] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 2.0
(TID 3) in 1717 ms on localhost (1/1)
21711 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 2 (first at
package.scala:113) finished in 1.718 s
21711 [task-result-getter-3] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 2.0, whose
tasks have all completed, from pool
21711 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 1
finished: first at package.scala:113, took 1.726375 s
22377 [main] INFO  com.spotify.spark.bigquery.BigQueryClient  - Executing
query SELECT word, word_count FROM
[bigquery-public-data:samples.shakespeare]
22666 [main] INFO  com.spotify.spark.bigquery.BigQueryClient  - Staging
dataset pure-respect-180709:spark_bigquery_staging_us already exists
22666 [main] INFO  com.spotify.spark.bigquery.BigQueryClient  - Destination
table: {datasetId=spark_bigquery_staging_us, projectId=pure-respect-180709,
tableId=spark_bigquery_20171004041401_1420172520}
159641 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using
specified project-id 'pure-respect-180709' for input
159643 [main] INFO  org.apache.spark.storage.MemoryStore  - Block
broadcast_5 stored as values in memory (estimated size 230.6 KB, free 735.0
KB)
159653 [main] INFO  org.apache.spark.storage.MemoryStore  - Block
broadcast_5_piece0 stored as bytes in memory (estimated size 22.5 KB, free
757.5 KB)
159654 [dispatcher-event-loop-0] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_5_piece0 in
memory on localhost:34778 (size: 22.5 KB, free: 4.1 GB)
159654 [main] INFO  org.apache.spark.SparkContext  - Created broadcast 5
from newAPIHadoopRDD at package.scala:111
159657 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from default credential.
159658 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from given credential.
159658 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Fetching key
'mapred.bq.gcs.bucket' since 'mapred.bq.temp.gcs.path' isn't set explicitly.
159659 [main] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration  - Using working
path: 'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008'
161147 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
161148 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'pure-respect-180709:spark_bigquery_staging_us.spark_bigquery_20171004041401_1420172520'
to be exported has 164656 rows and 2650191 bytes
161377 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Computed
'2' shards for sharded BigQuery export.
161378 [main] INFO
com.google.cloud.hadoop.io.bigquery.ShardedExportToCloudStorage  - Table
'pure-respect-180709:spark_bigquery_staging_us.spark_bigquery_20171004041401_1420172520'
to be exported has 164656 rows and 2650191 bytes
161383 [main] INFO  org.apache.spark.SparkContext  - Starting job: first at
package.scala:113
161384 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 2 (first at
package.scala:113) with 1 output partitions
161384 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 3
(first at package.scala:113)
161384 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage: List()
161384 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
161384 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 3
(MapPartitionsRDD[10] at map at package.scala:113), which has no missing
parents
161386 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_6 stored as values
in memory (estimated size 2.8 KB, free 760.3 KB)
161388 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_6_piece0 stored as
bytes in memory (estimated size 1756.0 B, free 762.0 KB)
161388 [dispatcher-event-loop-3] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_6_piece0 in
memory on localhost:34778 (size: 1756.0 B, free: 4.1 GB)
161389 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 6 from broadcast at DAGScheduler.scala:1006
161389 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 1 missing tasks from
ResultStage 3 (MapPartitionsRDD[10] at map at package.scala:113)
161389 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 3.0 with 1
tasks
161390 [dispatcher-event-loop-4] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 3.0
(TID 4, localhost, partition 0,PROCESS_LOCAL, 2303 bytes)
161391 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 3.0 (TID 4)
161393 [Executor task launch worker-2] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]
161393 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040016_0008_m_000000_0 Status:'
162366 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 1667061 to knownFileSet.
162367 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-000000000000.avro'
which has 1667061 bytes. Records read so far: 0
163070 [Executor task launch worker-2] WARN
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Got
non-null delegateReader during close(); possible premature close() call.
163072 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 3.0 (TID
4). 2230 bytes result sent to driver
163073 [task-result-getter-0] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 3.0
(TID 4) in 1683 ms on localhost (1/1)
163073 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 3 (first at
package.scala:113) finished in 1.684 s
163073 [task-result-getter-0] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 3.0, whose
tasks have all completed, from pool
163074 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 2
finished: first at package.scala:113, took 1.690461 s
Map(spark.sql.thriftServer.incrementalCollect -> true,
spark.sql.inMemoryColumnarStorage.compressed -> true,
spark.sql.inMemoryColumnarStorage.partitionPruning -> true)
163173 [main] INFO  org.apache.spark.SparkContext  - Starting job: take at
App.scala:116
163174 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 3 (take at
App.scala:116) with 1 output partitions
163174 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 4 (take
at App.scala:116)
163174 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage: List()
163175 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
163175 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 4
(MapPartitionsRDD[13] at take at App.scala:116), which has no missing
parents
163178 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_7 stored as values
in memory (estimated size 5.0 KB, free 766.9 KB)
163179 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_7_piece0 stored as
bytes in memory (estimated size 2.6 KB, free 769.6 KB)
163180 [dispatcher-event-loop-7] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_7_piece0 in
memory on localhost:34778 (size: 2.6 KB, free: 4.1 GB)
163181 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 7 from broadcast at DAGScheduler.scala:1006
163181 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 1 missing tasks from
ResultStage 4 (MapPartitionsRDD[13] at take at App.scala:116)
163181 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 4.0 with 1
tasks
163182 [dispatcher-event-loop-8] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 4.0
(TID 5, localhost, partition 0,PROCESS_LOCAL, 2303 bytes)
163183 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 4.0 (TID 5)
163189 [Executor task launch worker-2] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]
163190 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040016_0008_m_000000_0 Status:'
163686 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 1667061 to knownFileSet.
163686 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000001.avro' of size 409 to knownFileSet.
163687 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-000000000000.avro'
which has 1667061 bytes. Records read so far: 0
164410 [Executor task launch worker-2] WARN
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Got
non-null delegateReader during close(); possible premature close() call.
164417 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 4.0 (TID
5). 6022 bytes result sent to driver
164419 [task-result-getter-1] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 4.0
(TID 5) in 1237 ms on localhost (1/1)
164419 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 4 (take at
App.scala:116) finished in 1.238 s
164419 [task-result-getter-1] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 4.0, whose
tasks have all completed, from pool
164420 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 3
finished: take at App.scala:116, took 1.246298 s
164441 [main] INFO  org.apache.spark.SparkContext  - Starting job: take at
App.scala:125
164442 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 4 (take at
App.scala:125) with 1 output partitions
164442 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 5 (take
at App.scala:125)
164442 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage: List()
164442 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
164443 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 5
(MapPartitionsRDD[14] at toJSON at App.scala:125), which has no missing
parents
164444 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_8 stored as values
in memory (estimated size 5.3 KB, free 774.9 KB)
164445 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_8_piece0 stored as
bytes in memory (estimated size 2.8 KB, free 777.7 KB)
164446 [dispatcher-event-loop-11] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_8_piece0 in
memory on localhost:34778 (size: 2.8 KB, free: 4.1 GB)
164446 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 8 from broadcast at DAGScheduler.scala:1006
164446 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 1 missing tasks from
ResultStage 5 (MapPartitionsRDD[14] at toJSON at App.scala:125)
164446 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 5.0 with 1
tasks
164447 [dispatcher-event-loop-12] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 5.0
(TID 6, localhost, partition 0,PROCESS_LOCAL, 2303 bytes)
164447 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 5.0 (TID 6)
164450 [Executor task launch worker-2] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]
164451 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040016_0008_m_000000_0 Status:'
164866 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 1667061 to knownFileSet.
164866 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000001.avro' of size 409 to knownFileSet.
164866 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-000000000000.avro'
which has 1667061 bytes. Records read so far: 0
165361 [Executor task launch worker-2] WARN
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Got
non-null delegateReader during close(); possible premature close() call.
165362 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 5.0 (TID
6). 5168 bytes result sent to driver
165364 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 5 (take at
App.scala:125) finished in 0.917 s
165364 [task-result-getter-2] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 5.0
(TID 6) in 916 ms on localhost (1/1)
165364 [task-result-getter-2] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 5.0, whose
tasks have all completed, from pool
165364 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 4
finished: take at App.scala:125, took 0.923151 s
165712 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.job.id is deprecated. Instead, use mapreduce.job.id
165713 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.tip.id is deprecated. Instead, use mapreduce.task.id
165713 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
165713 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
165713 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
165722 [main] INFO  com.databricks.spark.avro.AvroRelation  - using
deflate: -1 for Avro output
165723 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
165724 [main] INFO
com.google.cloud.hadoop.io.bigquery.output.ForwardingBigQueryFileOutputFormat
Delegating functionality to 'TextOutputFormat'.
165746 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from default credential.
165747 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryFactory  -
Creating BigQuery from given credential.
165748 [main] INFO
org.apache.spark.sql.execution.datasources.DefaultWriterContainer  - Using
output committer class
com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputCommitter
166777 [main] INFO  org.apache.spark.SparkContext  - Starting job: save at
package.scala:37
166778 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Got job 5 (save at
package.scala:37) with 2 output partitions
166778 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Final stage: ResultStage 6 (save
at package.scala:37)
166779 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Parents of final stage: List()
166779 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Missing parents: List()
166779 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting ResultStage 6
(MapPartitionsRDD[12] at createDataFrame at package.scala:119), which has
no missing parents
166789 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_9 stored as values
in memory (estimated size 71.0 KB, free 848.7 KB)
166791 [dag-scheduler-event-loop] INFO
org.apache.spark.storage.MemoryStore  - Block broadcast_9_piece0 stored as
bytes in memory (estimated size 27.1 KB, free 875.7 KB)
166792 [dispatcher-event-loop-16] INFO
org.apache.spark.storage.BlockManagerInfo  - Added broadcast_9_piece0 in
memory on localhost:34778 (size: 27.1 KB, free: 4.1 GB)
166792 [dag-scheduler-event-loop] INFO  org.apache.spark.SparkContext  -
Created broadcast 9 from broadcast at DAGScheduler.scala:1006
166792 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - Submitting 2 missing tasks from
ResultStage 6 (MapPartitionsRDD[12] at createDataFrame at package.scala:119)
166792 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Adding task set 6.0 with 2
tasks
166793 [dispatcher-event-loop-15] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 0.0 in stage 6.0
(TID 7, localhost, partition 0,PROCESS_LOCAL, 2303 bytes)
166794 [dispatcher-event-loop-15] INFO
org.apache.spark.scheduler.TaskSetManager  - Starting task 1.0 in stage 6.0
(TID 8, localhost, partition 1,PROCESS_LOCAL, 2303 bytes)
166794 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Running task 0.0 in stage 6.0 (TID 7)
166795 [Executor task launch worker-3] INFO
org.apache.spark.executor.Executor  - Running task 1.0 in stage 6.0 (TID 8)
166815 [Executor task launch worker-2] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]
166816 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040016_0008_m_000000_0 Status:'
166817 [Executor task launch worker-3] INFO
org.apache.spark.rdd.NewHadoopRDD  - Input split:
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-1/data-.avro[82328
estimated records]
166817 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  -
Initializing DynamicFileListRecordReader with split 'InputSplit::
length:82328 locations: [] toString():
gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-1/data-.avro[82328
estimated records]', task context 'TaskAttemptContext::
TaskAttemptID:attempt_201710040016_0008_m_000001_0 Status:'
167042 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.output.ForwardingBigQueryFileOutputFormat
Delegating functionality to 'TextOutputFormat'.
167069 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryFactory  - Creating BigQuery
from default credential.
167071 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryFactory  - Creating BigQuery
from given credential.
167071 [Executor task launch worker-2] INFO
org.apache.spark.sql.execution.datasources.DefaultWriterContainer  - Using
output committer class
com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputCommitter
167419 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.output.ForwardingBigQueryFileOutputFormat
Delegating functionality to 'TextOutputFormat'.
167441 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryFactory  - Creating BigQuery
from default credential.
167441 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.BigQueryFactory  - Creating BigQuery
from given credential.
167747 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 1667061 to knownFileSet.
169086 [Executor task launch worker-3] INFO
org.apache.spark.sql.execution.datasources.DefaultWriterContainer  - Using
output committer class
com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputCommitter
169086 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000001.avro' of size 409 to knownFileSet.
169086 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-000000000000.avro'
which has 1667061 bytes. Records read so far: 0
169403 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 3
169403 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 4
169406 [dispatcher-event-loop-21] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_6_piece0 on
localhost:34778 in memory (size: 1756.0 B, free: 4.1 GB)
169409 [dispatcher-event-loop-24] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_7_piece0 on
localhost:34778 in memory (size: 2.6 KB, free: 4.1 GB)
169412 [dispatcher-event-loop-27] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_8_piece0 on
localhost:34778 in memory (size: 2.8 KB, free: 4.1 GB)
169413 [dispatcher-event-loop-30] INFO
org.apache.spark.storage.BlockManagerInfo  - Removed broadcast_4_piece0 on
localhost:34778 in memory (size: 1751.0 B, free: 4.1 GB)
169413 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 5
169413 [Spark Context Cleaner] INFO  org.apache.spark.ContextCleaner  -
Cleaned accumulator 6
169751 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Adding
new file 'data-000000000000.avro' of size 409 to knownFileSet.
169751 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-1/data-000000000000.avro'
which has 409 bytes. Records read so far: 0
170220 [Executor task launch worker-3] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Found
end-marker file 'data-000000000000.avro' with index 0
170664 [Executor task launch worker-3] INFO
org.apache.spark.mapred.SparkHadoopMapRedUtil  - No need to commit output
of task because needsTaskCommit=false: attempt_201710040016_0006_m_000001_0
170665 [Executor task launch worker-3] INFO
org.apache.spark.executor.Executor  - Finished task 1.0 in stage 6.0 (TID
8). 2099 bytes result sent to driver
170666 [task-result-getter-3] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 1.0 in stage 6.0
(TID 8) in 3872 ms on localhost (1/2)
171002 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Moving
to next file
'gs://iipbucket/hadoop/tmp/bigquery/job_201710040016_0008/shard-0/data-000000000001.avro'
which has 409 bytes. Records read so far: 164656
171390 [Executor task launch worker-2] INFO
com.google.cloud.hadoop.io.bigquery.DynamicFileListRecordReader  - Found
end-marker file 'data-000000000001.avro' with index 1
171947 [Executor task launch worker-2] INFO
org.apache.spark.mapred.SparkHadoopMapRedUtil  - No need to commit output
of task because needsTaskCommit=false: attempt_201710040016_0006_m_000000_0
171948 [Executor task launch worker-2] INFO
org.apache.spark.executor.Executor  - Finished task 0.0 in stage 6.0 (TID
7). 2099 bytes result sent to driver
171949 [task-result-getter-0] INFO
org.apache.spark.scheduler.TaskSetManager  - Finished task 0.0 in stage 6.0
(TID 7) in 5156 ms on localhost (2/2)
171949 [dag-scheduler-event-loop] INFO
org.apache.spark.scheduler.DAGScheduler  - ResultStage 6 (save at
package.scala:37) finished in 5.156 s
171949 [task-result-getter-0] INFO
org.apache.spark.scheduler.TaskSchedulerImpl  - Removed TaskSet 6.0, whose
tasks have all completed, from pool
171949 [main] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 5
finished: save at package.scala:37, took 5.171605 s
173738 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryHelper  -
Importing into table 'pure-respect-180709:wordcount_dataset.word_cnt' from
2 paths; path[0] is
'gs://iipbucket/hadoop/tmp/spark-bigquery/spark-bigquery-1507090584737=1777134178/part-r-00000-188a4569-2aa6-44d3-9683-a21dd437790a.avro';
awaitCompletion: true
173739 [main] INFO  com.google.cloud.hadoop.io.bigquery.BigQueryHelper  -
No import schema provided, auto detecting schema.
174366 [main] ERROR
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation  -
Aborting job.
java.io.IOException: Failed to parse JSON: Unexpected token; Parser
terminated before end of string
at
com.google.cloud.hadoop.io.bigquery.BigQueryUtils.waitForJobCompletion(BigQueryUtils.java:95)
at
com.google.cloud.hadoop.io.bigquery.BigQueryHelper.importFromGcs(BigQueryHelper.java:164)
at
com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputCommitter.commitJob(IndirectBigQueryOutputCommitter.java:57)
at
org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:149)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at
org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)
at
com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:37)
at
com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:37)
at
com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:150)
at
com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:162)
at bigquery.App$.main(App.scala:126)
at bigquery.App.main(App.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
174724 [main] INFO
com.google.cloud.hadoop.io.bigquery.output.ForwardingBigQueryFileOutputCommitter
Found GCS output data at
'gs://iipbucket/hadoop/tmp/spark-bigquery/spark-bigquery-1507090584737=1777134178',
attempting to clean up.
175349 [main] INFO
com.google.cloud.hadoop.io.bigquery.output.ForwardingBigQueryFileOutputCommitter
Successfully deleted GCS output path
'gs://iipbucket/hadoop/tmp/spark-bigquery/spark-bigquery-1507090584737=1777134178'.
175350 [main] ERROR
org.apache.spark.sql.execution.datasources.DefaultWriterContainer  - Job
job_201710040016_0000 aborted.
Exception in thread "main" org.apache.spark.SparkException: Job aborted.
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:154)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:106)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at
org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)
at
com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:37)
at
com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:37)
at
com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:150)
at
com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:162)
at bigquery.App$.main(App.scala:126)
at bigquery.App.main(App.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Failed to parse JSON: Unexpected token;
Parser terminated before end of string
at
com.google.cloud.hadoop.io.bigquery.BigQueryUtils.waitForJobCompletion(BigQueryUtils.java:95)
at
com.google.cloud.hadoop.io.bigquery.BigQueryHelper.importFromGcs(BigQueryHelper.java:164)
at
com.google.cloud.hadoop.io.bigquery.output.IndirectBigQueryOutputCommitter.commitJob(IndirectBigQueryOutputCommitter.java:57)
at
org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:149)
... 31 more
175353 [Thread-3] INFO  org.apache.spark.SparkContext  - Invoking stop()
from shutdown hook
175437 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/static/sql,null}
175438 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/SQL/execution/json,null}
175438 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/SQL/execution,null}
175438 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/SQL/json,null}
175439 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/SQL,null}
175439 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
175439 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
175439 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/api,null}
175439 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/static,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/executors,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
175440 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/environment,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/storage,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
175441 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/stages,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
175442 [Thread-3] INFO
org.spark-project.jetty.server.handler.ContextHandler  - stopped
o.s.j.s.ServletContextHandler{/jobs,null}
175495 [Thread-3] INFO  org.apache.spark.ui.SparkUI  - Stopped Spark web UI
at http://<IP_ADDRESS>:8085
175520 [dispatcher-event-loop-5] INFO
org.apache.spark.MapOutputTrackerMasterEndpoint  -
MapOutputTrackerMasterEndpoint stopped!
175526 [Thread-3] INFO  org.apache.spark.storage.MemoryStore  - MemoryStore
cleared
175527 [Thread-3] INFO  org.apache.spark.storage.BlockManager  -
BlockManager stopped
175528 [Thread-3] INFO  org.apache.spark.storage.BlockManagerMaster  -
BlockManagerMaster stopped
175529 [dispatcher-event-loop-10] INFO
org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint
OutputCommitCoordinator stopped!
175532 [Thread-3] INFO  org.apache.spark.SparkContext  - Successfully
stopped SparkContext
175534 [Thread-3] INFO  org.apache.spark.util.ShutdownHookManager  -
Shutdown hook called
175535 [Thread-3] INFO  org.apache.spark.util.ShutdownHookManager  -
Deleting directory
/mydata/Mana_2.3/tmp/spark-7bb14363-a18e-453c-a00c-3518c171a6ee
175537 [sparkDriverActorSystem-akka.actor.default-dispatcher-4] INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator  - Shutting down
remote daemon.
175538 [sparkDriverActorSystem-akka.actor.default-dispatcher-4] INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator  - Remote daemon shut
down; proceeding with flushing remote transports.
175558 [sparkDriverActorSystem-akka.actor.default-dispatcher-4] INFO
akka.remote.RemoteActorRefProvider$RemotingTerminator  - Remoting shut down.
175564 [Thread-3] INFO  org.apache.spark.util.ShutdownHookManager  -
Deleting directory
/mydata/Mana_2.3/tmp/spark-7bb14363-a18e-453c-a00c-3518c171a6ee/httpd-a83d59a5-55c0-4004-94e0-5f7409d187c4

On Wed, Oct 4, 2017 at 12:05 AM, Sam Elamin<EMAIL_ADDRESS>wrote:

not sure where the problem is here, can you post the entire stacktrace?
id seperate the reading vs writing to identify where the issue is
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/45#issuecomment-333937981,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALmGApE4dpOCLnYQAp_h6iDUm0YiU4g4ks5son6KgaJpZM4PsDns
.

Hi Daryl,
Are you talking about the auth  key file? That key file is of json format
and is generated directly from bigquery webui
On Tue, Oct 3, 2017 at 11:25 PM, Daryl Erwin<EMAIL_ADDRESS>wrote:

Appears the code saves the data as json then writes that into bigquery.
Are you able to review the json file for any issues?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/45#issuecomment-333926244,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALmGAhvPtbVbPaSEUD43x59Bwszjjbwiks5sonUPgaJpZM4PsDns
.

We only need the stack trace not the entire log of your application
But according to that stack trace it seems you might be using the spotify
version of the connector. Pr atleast you have it somewhere on the path
Can you make sure you have no reference to the spotify connector
On Wed, 4 Oct 2017 at 05:23, Ayyappadas1<EMAIL_ADDRESS>wrote:

Hi Daryl,
Are you talking about the auth key file? That key file is of json format
and is generated directly from bigquery webui
On Tue, Oct 3, 2017 at 11:25 PM, Daryl Erwin<EMAIL_ADDRESS>wrote:

Appears the code saves the data as json then writes that into bigquery.
Are you able to review the json file for any issues?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/samelamin/spark-bigquery/issues/45#issuecomment-333926244
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ALmGAhvPtbVbPaSEUD43x59Bwszjjbwiks5sonUPgaJpZM4PsDns
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/samelamin/spark-bigquery/issues/45#issuecomment-334046317,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHLm0FCCevQYB0QGXT1OYysRYApJU1Sks5sowgzgaJpZM4PsDns
.