feat(risedev): support configuring meta store by url from env var

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
See ideas in #19527.
When specifying meta-backend: env for the meta-node service in risedev dev, RiseDev will now pass it verbatim as --sql-endpoint to the meta node.
Additionally, if the cluster is started with ci-start, RiseDev will help to recreate the database that will be used.
Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

yaml not updated? :eyes:

yaml not updated? 👀

Won't be in this PR. 🥰

#19560  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Suggestion: disable hw acceleration by default

Description
It appears from https://discuss.atom.io/t/why-is-atom-so-slow/11376/209 that a lot of people have bad performance from the start due to hw acceleration issues. While this is not as bad as crashes and such (which are probably already taken care of by chromium's built-in black list of really bad drivers) this still gives people a bad first experience with the editor. Since I assume with Atom's barely changing UI in comparison to like, a complex dynamic rendered 3d world, the hw acceleration doesn't do terribly much anyway, I suggest reconsidering whether it should be enabled by default at all.

Since I assume with Atom's barely changing UI...the hw acceleration doesn't do terribly much

This may come as a surprise to you but editor scrolling is entirely implemented using the GPU.  We use the transform: translateY(amount) property which signals to Chrome to use hardware acceleration.  You can test this yourself in Windows Fall Creators Update (and above) by opening the Task Manager, enabling the GPU column, and then looking at the value change when you scroll an editor in Atom.

Therefore it seems like hardware acceleration is fairly important to a smooth scrolling experience.  I'll defer to someone more experienced in this area like @nathansobo though in case I got something wrong.

Thanks for the suggestion. As @50Wliu pointed out, Atom does use the hardware acceleration quite a bit. Additionally, disabling hardware acceleration across the board will have an extremely negative performance impact for many people. So this is not something that we'll be implementing anytime soon.


added simple form challenge - beta version - under dev

Pull Request Template
Description
Describe your changes in detail
Related Issue
Reference the related issue(s) here, if applicable. For example, "Fixes #123" or "Addresses #456"
Proposed Changes
Please describe the changes you've made. Include details about the problem you are solving and the approach you took to address it.
Screenshots (if applicable)
Add screenshots to help explain your changes.

Reviewer(s) Requested

sorry i supposed to push the security branch not the main


writeGlobalFile probably needs a block if the input is a not-local file and won't be put into cache

I have a test pipeline as follows
F1 = Job.wrapJobFn(aux.write_to_filestore, self.fastqs)
F2 = Job.wrapJobFn(aux.write_to_filestore, self.star_options)
G = Job.wrapJobFn(PI.get_files_from_filestore, F2.rv(), self.test_dir)
F1.addChild(F2)
F2.addChild(G)
Job.Runner.startToil(F1, params)

F1 writes two 4.5G fastqs to the jobstore
F2 writes a 24G tar.gz file to the jobstore
G in this example will attempt to retrieve the index.tar.gz from jobstore and untar it.
aux.write_to_filestore is
def write_to_filestore(job, files):
    '''
    This will write a dict of files to the write_to_filestore.
    The input FILES object is expected to be a dict of
    key = filename or sample_type
    value = filepath if key == filename
            list of filepaths if key == sample_type
    Eg. {'SRR123.fastq': '/data/SRR123.fastq',
         'sample1': ['/data/s_1.fastq', '/data/s_2.fastq']}

    FILES will be updated to
    key = filename or sample_type
    value = jobStoreFileID if key == filename
            list of jobStoreFileIDs if key == sample_type
    '''
    for name in files.keys():
        #  if list, replace the items with the filestoreID for each item
        #  obtained from writeGlobalFile
        if isinstance(files[name], list):
            files[name] = [job.fileStore.writeGlobalFile(x) for x in files[name]]
        else:
            files[name] = job.fileStore.writeGlobalFile(files[name])
    return files

PI.get_files_from_filestore is
def get_files_from_filestore(job, files, work_dir, docker=False):
    """
    This is adapted from John Vivian's return_input_paths from the RNA-Seq
    pipeline.

    Returns the paths of files from the FileStore if they are not present.
    if docker=True, return the docker path for the file
    if the file extension is tar.gz, then tar -zxvf it
    files is a dict with
        keys = the name of the file to be returned in toil space
        value = the input value for the file (can be toil temp file)
    """
    for name in files.keys():
        files[name] = job.fileStore.readGlobalFile(files[name],
                                                   '/'.join([work_dir, name]))
        #  If the file pointed to a tarball, extract it to WORK_DIR
        if tarfile.is_tarfile(files[name]):
            files[name] = untargz(files[name], work_dir)
        #  If the files will be sent to docker, we will mount work_dir to the
        #  container as /data and we want the /data prefixed path to the file
        if docker == True:
            files[name] = docker_path(files[name])
    return files

def untargz(input_targz_file, untar_to_dir):
    '''
    This module accepts a tar.gz archive and untars it.

    RETURN VALUE: path to the untar-ed directory/file

    NOTE: this module expects the multiple files to be in a directory before
          being tar-ed.
    '''
    assert tarfile.is_tarfile(input_targz_file), 'Not a tar file.'
    tarball = tarfile.open(input_targz_file)
    return_value = os.path.join(untar_to_dir, tarball.getmembers()[0].name)
    tarball.extractall(path=untar_to_dir)
    tarball.close()
    return return_value

If i comment out the F2.addChild(G) line then the data is successfully written to the filestore and the program exits normally.
If i don't, I get this Error almost immediately
[centos@arjun-fake-podk toil_tests]$ python module_tests.py
No handlers could be found for logger "toil.resource"
INFO:toil.lib.bioio:Logging set at level: DEBUG
INFO:toil.lib.bioio:Logging to file: /data1/star_log
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
DEBUG:toil.batchSystems.singleMachine:Issuing the command: /usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir e/j/jobaSjV4A with memory:<PHONE_NUMBER>, cores: 1, disk:<PHONE_NUMBER>
DEBUG:toil.leader:Issued job with job store ID: e/j/jobaSjV4A and job batch system ID: 0 and cores: 1, disk:<PHONE_NUMBER>, and memory:<PHONE_NUMBER>
DEBUG:toil.batchSystems.singleMachine:Acquiring<PHONE_NUMBER> bytes of memory from a pool of<PHONE_NUMBER>854775807.
DEBUG:toil.batchSystems.singleMachine:Acquiring 10 fractional cores from a pool of 80 to satisfy a request of 1.000000 cores
INFO:toil.batchSystems.singleMachine:Executing command: '/usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir e/j/jobaSjV4A'.
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.job.FileStore object at 0x7f5142899f90>> ignored

I assume that a block on the write will prevent this from happening. I'm going to place a 10 minute sleep in F2 to allow the write to complete and rerun this. I'm putting this up for now anyway.

Oops, yes. Bugger - this is what happens if you try to do ten things at
once. I agree, the simple thing is to just block for files in the temporary
store.
On Fri, Oct 9, 2015 at 3:46 PM, Arjun Arkal Rao<EMAIL_ADDRESS>wrote:

I have a test pipeline as follows
F1 = Job.wrapJobFn(aux.write_to_filestore, self.fastqs)
F2 = Job.wrapJobFn(aux.write_to_filestore, self.star_options)
G = Job.wrapJobFn(PI.get_files_from_filestore, F2.rv(), self.test_dir)
F1.addChild(F2)
F2.addChild(G)
Job.Runner.startToil(F1, params)
aux.write_to_filestore is
def write_to_filestore(job, files):
'''
This will write a dict of files to the write_to_filestore.
The input FILES object is expected to be a dict of
key = filename or sample_type
value = filepath if key == filename
list of filepaths if key == sample_type
Eg. {'SRR123.fastq': '/data/SRR123.fastq',
'sample1': ['/data/s_1.fastq', '/data/s_2.fastq']}
FILES will be updated to
key = filename or sample_type
value = jobStoreFileID if key == filename
        list of jobStoreFileIDs if key == sample_type
'''
for name in files.keys():
    #  if list, replace the items with the filestoreID for each item
    #  obtained from writeGlobalFile
    if isinstance(files[name], list):
        files[name] = [job.fileStore.writeGlobalFile(x) for x in files[name]]
    else:
        files[name] = job.fileStore.writeGlobalFile(files[name])
return files

PI.get_files_from_filestore is
def get_files_from_filestore(job, files, work_dir, docker=False):
"""
This is adapted from John Vivian's return_input_paths from the RNA-Seq
pipeline.
Returns the paths of files from the FileStore if they are not present.
if docker=True, return the docker path for the file
if the file extension is tar.gz, then tar -zxvf it
files is a dict with
    keys = the name of the file to be returned in toil space
    value = the input value for the file (can be toil temp file)
"""
for name in files.keys():
    files[name] = job.fileStore.readGlobalFile(files[name],
                                               '/'.join([work_dir, name]))
    #  If the file pointed to a tarball, extract it to WORK_DIR
    if tarfile.is_tarfile(files[name]):
        files[name] = untargz(files[name], work_dir)
    #  If the files will be sent to docker, we will mount work_dir to the
    #  container as /data and we want the /data prefixed path to the file
    if docker == True:
        files[name] = docker_path(files[name])
return files

—
Reply to this email directly or view it on GitHub
https://github.com/BD2KGenomics/toil/issues/484.

Doesn't look like it's a blocking problem on the write. I ran it on a 1.5G tarball with a 15s sleep function
def beddybye(job, sleeptime):
    '''
    This will sleep for SLEEPTIME seconds. It also creates a temp working
    directory that it returns
    '''
    work_dir = job.fileStore.getLocalTempDir()
    with open('/data/star_logfile', 'w') as logfile:
        print('My workdir is ', work_dir, file=logfile)
    sleep(sleeptime)
    return work_dir

that returns a temp working directory. The main script is modified to
F2 = Job.wrapJobFn(aux.write_to_filestore, self.star_options)
S = Job.wrapJobFn(aux.beddybye, 15)
H = Job.wrapJobFn(PI.get_files_from_filestore, F2.rv(), S.rv())

F2.addChild(S)
S.addChild(H)

Job.Runner.startToil(F2, params)

So TL;DR - write a 1.5G bam into jobstore, wait 15s (ample time to write the entire file to jobstore), then attempt to read it back from filestore. This is the trace ( --logDebug)
[centos@arjun-fake-podk toil_tests]$ python module_tests.py
No handlers could be found for logger "toil.resource"
INFO:toil.lib.bioio:Logging set at level: DEBUG
INFO:toil.lib.bioio:Logging to file: /data1/star_log
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
DEBUG:toil.batchSystems.singleMachine:Issuing the command: /usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir 0/c/job7pbt0o with memory:<PHONE_NUMBER>, cores: 1, disk:<PHONE_NUMBER>
DEBUG:toil.leader:Issued job with job store ID: 0/c/job7pbt0o and job batch system ID: 0 and cores: 1, disk:<PHONE_NUMBER>, and memory:<PHONE_NUMBER>
DEBUG:toil.batchSystems.singleMachine:Acquiring<PHONE_NUMBER> bytes of memory from a pool of<PHONE_NUMBER>854775807.
DEBUG:toil.batchSystems.singleMachine:Acquiring 10 fractional cores from a pool of 80 to satisfy a request of 1.000000 cores
INFO:toil.batchSystems.singleMachine:Executing command: '/usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir 0/c/job7pbt0o'.
DEBUG:toil.batchSystems.singleMachine:Finished job. self.coreFractions ~ 80 and self.memory ~<PHONE_NUMBER>854775807
DEBUG:toil.batchSystems.singleMachine:Ran jobID: 0 with exit value: 0
DEBUG:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 0 and jobWrapper store ID: 0/c/job7pbt0o ended successfully
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 0/c/job7pbt0o
WARNING:toil.leader:Reporting file: 0/c/job7pbt0o
WARNING:toil.leader:0/c/job7pbt0o:	---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:0/c/job7pbt0o:	Next available file descriptor: 8
WARNING:toil.leader:0/c/job7pbt0o:	DEBUG:__main__:Next available file descriptor: 8
WARNING:toil.leader:0/c/job7pbt0o:	Parsed jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	DEBUG:__main__:Parsed jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:0/c/job7pbt0o:	Starting the next jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:0/c/job7pbt0o:	Starting the next jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:0/c/job7pbt0o:	INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/docker_scripts/precision_immuno.pyc'
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/docker_scripts', name='precision_immuno', extension='.pyc')
WARNING:toil.leader:0/c/job7pbt0o:	Traceback (most recent call last):
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 284, in main
WARNING:toil.leader:0/c/job7pbt0o:	    fileStore=fileStore)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1104, in _execute
WARNING:toil.leader:0/c/job7pbt0o:	    returnValues = self.run(fileStore)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1218, in run
WARNING:toil.leader:0/c/job7pbt0o:	    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/data/test_docker/docker_scripts/precision_immuno.py", line 1100, in get_files_from_filestore
WARNING:toil.leader:0/c/job7pbt0o:	    '/'.join([work_dir, name]))
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 528, in readGlobalFile
WARNING:toil.leader:0/c/job7pbt0o:	    self.jobStore.readFile(fileStoreID, localFilePath)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 167, in readFile
WARNING:toil.leader:0/c/job7pbt0o:	    shutil.copyfile(self._getAbsPath(jobStoreFileID), localFilePath)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/shutil.py", line 83, in copyfile
WARNING:toil.leader:0/c/job7pbt0o:	    with open(dst, 'wb') as fdst:
WARNING:toil.leader:0/c/job7pbt0o:	IOError: [Errno 2] No such file or directory: '/data1/temp_dir/tmp_FPCRNmnBcY/localTempDir/t8wTgwR/index_tar'
WARNING:toil.leader:0/c/job7pbt0o:	Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:0/c/job7pbt0o:	ERROR:__main__:Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:0/c/job7pbt0o:	WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 0/c/job7pbt0o to 0
WARNING:toil.leader:0/c/job7pbt0o:	Exception in thread Thread-13:
WARNING:toil.leader:0/c/job7pbt0o:	Traceback (most recent call last):
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
WARNING:toil.leader:0/c/job7pbt0o:	    self.run()
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/threading.py", line 763, in run
WARNING:toil.leader:0/c/job7pbt0o:	    self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
WARNING:toil.leader:0/c/job7pbt0o:	    raise RuntimeError("The termination flag is set, exiting")
WARNING:toil.leader:0/c/job7pbt0o:	RuntimeError: The termination flag is set, exiting
WARNING:toil.leader:0/c/job7pbt0o:
WARNING:toil.leader:0/c/job7pbt0o:	Exception in thread Thread-14:
WARNING:toil.leader:0/c/job7pbt0o:	Traceback (most recent call last):
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
WARNING:toil.leader:0/c/job7pbt0o:	    self.run()
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/threading.py", line 763, in run
WARNING:toil.leader:0/c/job7pbt0o:	    self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:0/c/job7pbt0o:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
WARNING:toil.leader:0/c/job7pbt0o:	    raise RuntimeError("The termination flag is set, exiting")
WARNING:toil.leader:0/c/job7pbt0o:	RuntimeError: The termination flag is set, exiting
WARNING:toil.leader:0/c/job7pbt0o:
DEBUG:toil.leader:Added jobWrapper: 0/c/job7pbt0o to active jobs
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
WARNING:toil.leader:Job: 0/c/job7pbt0o is completely failed
INFO:toil.leader:Only failed jobs and their dependents (1 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish
INFO:toil.leader:Stats/logging finished collating in 0.0275518894196 seconds
DEBUG:toil.common:Shutting down batch system
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
Traceback (most recent call last):
  File "module_tests.py", line 210, in <module>
    A.test_star()
  File "module_tests.py", line 107, in test_star
    Job.Runner.startToil(F2, params)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 362, in startToil
    return mainLoop(config, batchSystem, jobStore, rootJob)
  File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 505, in mainLoop
    raise FailedJobsException( config.jobStore, totalFailedJobs )
toil.leader.FailedJobsException: The job store '/data1/toil_tempdir' contains 1 failed jobs

I honestly don't understand the IOError: [Errno 2] No such file or directory: '/data1/temp_dir/tmp_FPCRNmnBcY/localTempDir/t8wTgwR/index_tar' error since the temp directory exists and star_logifle confirms it.
I also don't understand why the thread errors are coming after this OSError since I can see from watch tree-fh /data1/toil_tempdir that the entire file is already in the filestore. In my previous trace, the thread errors were before the OSError.
[centos@arjun-fake-podk toil_tests]$ python module_tests.py
No handlers could be found for logger "toil.resource"
INFO:toil.lib.bioio:Logging set at level: DEBUG
INFO:toil.lib.bioio:Logging to file: /data1/star_log
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
DEBUG:toil.batchSystems.singleMachine:Issuing the command: /usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir r/4/jobP1dGoT with memory:<PHONE_NUMBER>, cores: 1, disk:<PHONE_NUMBER>
DEBUG:toil.leader:Issued job with job store ID: r/4/jobP1dGoT and job batch system ID: 0 and cores: 1, disk:<PHONE_NUMBER>, and memory:<PHONE_NUMBER>
DEBUG:toil.batchSystems.singleMachine:Acquiring<PHONE_NUMBER> bytes of memory from a pool of<PHONE_NUMBER>854775807.
DEBUG:toil.batchSystems.singleMachine:Acquiring 10 fractional cores from a pool of 80 to satisfy a request of 1.000000 cores
INFO:toil.batchSystems.singleMachine:Executing command: '/usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir r/4/jobP1dGoT'.
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 600, in asyncUpdate
    raise RuntimeError("The termination flag is set, exiting before update")
RuntimeError: The termination flag is set, exiting before update

Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 600, in asyncUpdate
    raise RuntimeError("The termination flag is set, exiting before update")
RuntimeError: The termination flag is set, exiting before update

Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.job.FileStore object at 0x7f9de4d2cc10>> ignored
DEBUG:toil.batchSystems.singleMachine:Finished job. self.coreFractions ~ 80 and self.memory ~<PHONE_NUMBER>854775807
DEBUG:toil.batchSystems.singleMachine:Ran jobID: 0 with exit value: 0
DEBUG:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 0 and jobWrapper store ID: r/4/jobP1dGoT ended successfully
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: r/4/jobP1dGoT
WARNING:toil.leader:Reporting file: r/4/jobP1dGoT
WARNING:toil.leader:r/4/jobP1dGoT:	---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:r/4/jobP1dGoT:	Next available file descriptor: 8
WARNING:toil.leader:r/4/jobP1dGoT:	DEBUG:__main__:Next available file descriptor: 8
WARNING:toil.leader:r/4/jobP1dGoT:	Parsed jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	DEBUG:__main__:Parsed jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:r/4/jobP1dGoT:	Starting the next jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:r/4/jobP1dGoT:	Starting the next jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:r/4/jobP1dGoT:	INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/docker_scripts/precision_immuno.pyc'
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/docker_scripts', name='precision_immuno', extension='.pyc')
WARNING:toil.leader:r/4/jobP1dGoT:	Traceback (most recent call last):
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 284, in main
WARNING:toil.leader:r/4/jobP1dGoT:	    fileStore=fileStore)
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1104, in _execute
WARNING:toil.leader:r/4/jobP1dGoT:	    returnValues = self.run(fileStore)
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1218, in run
WARNING:toil.leader:r/4/jobP1dGoT:	    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/data/test_docker/docker_scripts/precision_immuno.py", line 1100, in get_files_from_filestore
WARNING:toil.leader:r/4/jobP1dGoT:	    '/'.join([work_dir, name]))
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 528, in readGlobalFile
WARNING:toil.leader:r/4/jobP1dGoT:	    self.jobStore.readFile(fileStoreID, localFilePath)
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 167, in readFile
WARNING:toil.leader:r/4/jobP1dGoT:	    shutil.copyfile(self._getAbsPath(jobStoreFileID), localFilePath)
WARNING:toil.leader:r/4/jobP1dGoT:	  File "/usr/local/lib/python2.7/shutil.py", line 83, in copyfile
WARNING:toil.leader:r/4/jobP1dGoT:	    with open(dst, 'wb') as fdst:
WARNING:toil.leader:r/4/jobP1dGoT:	IOError: [Errno 2] No such file or directory: '/data1/temp_dir/tmp_RMz7RGIgJx/localTempDir/tJSFLJM/index_tar'
WARNING:toil.leader:r/4/jobP1dGoT:	Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:r/4/jobP1dGoT:	ERROR:__main__:Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:r/4/jobP1dGoT:	WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job r/4/jobP1dGoT to 0
DEBUG:toil.leader:Added jobWrapper: r/4/jobP1dGoT to active jobs
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
WARNING:toil.leader:Job: r/4/jobP1dGoT is completely failed
INFO:toil.leader:Only failed jobs and their dependents (1 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish
INFO:toil.leader:Stats/logging finished collating in 0.399203062057 seconds
DEBUG:toil.common:Shutting down batch system
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
Traceback (most recent call last):
  File "module_tests.py", line 210, in <module>
    A.test_star()
  File "module_tests.py", line 107, in test_star
    Job.Runner.startToil(F2, params)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 362, in startToil
    return mainLoop(config, batchSystem, jobStore, rootJob)
  File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 505, in mainLoop
    raise FailedJobsException( config.jobStore, totalFailedJobs )
toil.leader.FailedJobsException: The job store '/data1/toil_tempdir' contains 1 failed jobs

My guess is that the OSError is being thrown because the failed threads cause the program to crash and localTempDir to get deleted before the untargz is called hence the folder 'doesn't exist'. What is weird is that the threads crash but the tarfile in the jobstore extracts without error (when i manually attempt it) so there's no data corruption. So this begs teh question, why are those threads erroring out? Increasing the wait time to 10 minutes (way wayyyyy more than required to write the file) changes nothing. There are no errors thrown till the next job starts and the error is the same.
[centos@arjun-fake-podk toil_tests]$ python module_tests.py
No handlers could be found for logger "toil.resource"
INFO:toil.lib.bioio:Logging set at level: DEBUG
INFO:toil.lib.bioio:Logging to file: /data1/star_log
INFO:toil.common:Using the single machine batch system
INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (8).
INFO:toil.batchSystems.singleMachine:Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
INFO:toil.common:Written the environment for the jobs to the environment file
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/site-packages', name='toil.job', extension='.pyc').
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
DEBUG:toil.batchSystems.singleMachine:Issuing the command: /usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir l/J/jobcMf1Cx with memory:<PHONE_NUMBER>, cores: 1, disk:<PHONE_NUMBER>
DEBUG:toil.leader:Issued job with job store ID: l/J/jobcMf1Cx and job batch system ID: 0 and cores: 1, disk:<PHONE_NUMBER>, and memory:<PHONE_NUMBER>
DEBUG:toil.batchSystems.singleMachine:Acquiring<PHONE_NUMBER> bytes of memory from a pool of<PHONE_NUMBER>854775807.
DEBUG:toil.batchSystems.singleMachine:Acquiring 10 fractional cores from a pool of 80 to satisfy a request of 1.000000 cores
INFO:toil.batchSystems.singleMachine:Executing command: '/usr/local/bin/python -E /usr/local/lib/python2.7/site-packages/toil/worker.py /data1/toil_tempdir l/J/jobcMf1Cx'.
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 398, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.job.FileStore object at 0x7fbe7835cc10>> ignored
DEBUG:toil.batchSystems.singleMachine:Finished job. self.coreFractions ~ 80 and self.memory ~<PHONE_NUMBER>854775807
DEBUG:toil.batchSystems.singleMachine:Ran jobID: 0 with exit value: 0
DEBUG:toil.leader:Batch system is reporting that the jobWrapper with batch system ID: 0 and jobWrapper store ID: l/J/jobcMf1Cx ended successfully
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: l/J/jobcMf1Cx
WARNING:toil.leader:Reporting file: l/J/jobcMf1Cx
WARNING:toil.leader:l/J/jobcMf1Cx:	---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:l/J/jobcMf1Cx:	Next available file descriptor: 8
WARNING:toil.leader:l/J/jobcMf1Cx:	DEBUG:__main__:Next available file descriptor: 8
WARNING:toil.leader:l/J/jobcMf1Cx:	Parsed jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	DEBUG:__main__:Parsed jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:l/J/jobcMf1Cx:	Starting the next jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/toil_tests/aux_scripts.pyc'
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/toil_tests', name='aux_scripts', extension='.pyc')
WARNING:toil.leader:l/J/jobcMf1Cx:	Starting the next jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	DEBUG:__main__:Starting the next jobWrapper
WARNING:toil.leader:l/J/jobcMf1Cx:	INFO:toil.jobStores.fileJobStore:Jobstore directory is: /data1/toil_tempdir
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't find resource for leader path '/data/test_docker/docker_scripts/precision_immuno.pyc'
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/data/test_docker/docker_scripts', name='precision_immuno', extension='.pyc')
WARNING:toil.leader:l/J/jobcMf1Cx:	Traceback (most recent call last):
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 284, in main
WARNING:toil.leader:l/J/jobcMf1Cx:	    fileStore=fileStore)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1104, in _execute
WARNING:toil.leader:l/J/jobcMf1Cx:	    returnValues = self.run(fileStore)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1218, in run
WARNING:toil.leader:l/J/jobcMf1Cx:	    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/data/test_docker/docker_scripts/precision_immuno.py", line 1105, in get_files_from_filestore
WARNING:toil.leader:l/J/jobcMf1Cx:	    '/'.join([work_dir, name]))
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 528, in readGlobalFile
WARNING:toil.leader:l/J/jobcMf1Cx:	    self.jobStore.readFile(fileStoreID, localFilePath)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 167, in readFile
WARNING:toil.leader:l/J/jobcMf1Cx:	    shutil.copyfile(self._getAbsPath(jobStoreFileID), localFilePath)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/shutil.py", line 83, in copyfile
WARNING:toil.leader:l/J/jobcMf1Cx:	    with open(dst, 'wb') as fdst:
WARNING:toil.leader:l/J/jobcMf1Cx:	IOError: [Errno 2] No such file or directory: '/data1/temp_dir/tmp_PW74NwRRLa/localTempDir/t0OuPSi/index_tar'
WARNING:toil.leader:l/J/jobcMf1Cx:	Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:l/J/jobcMf1Cx:	ERROR:__main__:Exiting the worker because of a failed jobWrapper on host arjun-fake-podk.novalocal
WARNING:toil.leader:l/J/jobcMf1Cx:	WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job l/J/jobcMf1Cx to 0
WARNING:toil.leader:l/J/jobcMf1Cx:	Exception in thread Thread-9:
WARNING:toil.leader:l/J/jobcMf1Cx:	Traceback (most recent call last):
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
WARNING:toil.leader:l/J/jobcMf1Cx:	    self.run()
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/threading.py", line 763, in run
WARNING:toil.leader:l/J/jobcMf1Cx:	    self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 600, in asyncUpdate
WARNING:toil.leader:l/J/jobcMf1Cx:	    raise RuntimeError("The termination flag is set, exiting before update")
WARNING:toil.leader:l/J/jobcMf1Cx:	RuntimeError: The termination flag is set, exiting before update
WARNING:toil.leader:l/J/jobcMf1Cx:
WARNING:toil.leader:l/J/jobcMf1Cx:	Exception in thread Thread-12:
WARNING:toil.leader:l/J/jobcMf1Cx:	Traceback (most recent call last):
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
WARNING:toil.leader:l/J/jobcMf1Cx:	    self.run()
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/threading.py", line 763, in run
WARNING:toil.leader:l/J/jobcMf1Cx:	    self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:l/J/jobcMf1Cx:	  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 600, in asyncUpdate
WARNING:toil.leader:l/J/jobcMf1Cx:	    raise RuntimeError("The termination flag is set, exiting before update")
WARNING:toil.leader:l/J/jobcMf1Cx:	RuntimeError: The termination flag is set, exiting before update
WARNING:toil.leader:l/J/jobcMf1Cx:
DEBUG:toil.leader:Added jobWrapper: l/J/jobcMf1Cx to active jobs
DEBUG:toil.leader:Built the jobs list, currently have 1 jobs to update and 0 jobs issued
WARNING:toil.leader:Job: l/J/jobcMf1Cx is completely failed
INFO:toil.leader:Only failed jobs and their dependents (1 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish
INFO:toil.leader:Stats/logging finished collating in 0.489140987396 seconds
DEBUG:toil.common:Shutting down batch system
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Received queue sentinel.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
DEBUG:toil.batchSystems.singleMachine:Exiting worker thread normally.
Traceback (most recent call last):
  File "module_tests.py", line 210, in <module>
    A.test_star()
  File "module_tests.py", line 107, in test_star
    Job.Runner.startToil(F2, params)
  File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 362, in startToil
    return mainLoop(config, batchSystem, jobStore, rootJob)
  File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 505, in mainLoop
    raise FailedJobsException( config.jobStore, totalFailedJobs )
toil.leader.FailedJobsException: The job store '/data1/toil_tempdir' contains 1 failed jobs

Arjun, John, this is now fixed, I've also made readGlobalFile more flexible, which I think you'll like.

The changes look great. Thanks Benedict. Do we know when this will be merged?

If in a hurry, @arkal you can always checkout the source branch of a PR without having to wait for it to be merged.


[Feature] Treesitter powered formatting

As mentioned in #162 and some previous issues, it would be great if we could use the appropriate formatter for the filetype that treesitter detects.
Example of usage would be a markdown file with code blocks in lua or any other language. formatter.nvim should format the markdown using something like prettier and other languages, lets say lua, using their configured formatters, or in case of lua using something like stylua.
There are a couple of concerns here though, and the main one is indentation. If the code inside the code block is indented the code block formatter would not have any knowledge that it is formatting an indented code block and, therefore, would mess up indentation of the code block. This could be solved by detecting the indentation before formatting and indenting the code after a formatter has finished formatting it.
Another concern is performance since there could be many code blocks in one file and formatter.nvim would have to run a formatter for each of them. This could be remedied by runninng them in parallel and/or somehow running all code blocks of the same language in one go.
This would require a lot more discussion and development, so I'm inviting everyone to join in on the conversation!

I wrote a CLI tool that does exactly this https://github.com/lukas-reineke/cbfmt
It uses treesitter to find the codeblocks and formats them with any tool you want, it handles indentation, and it is parallel.
You should be able to just use it with formatter.nvim as well

@lukas-reineke That is very nice! I'm not sure if you're familiar with the AUR, but can I add a package for it?
I have a couple of problems with this, though, and it's that you would have to specify formatters twice: first time for nvim.formatter and the second time for cbfmt. The second problem is that it wouldn't solve range based formatting and especially for files that have multiple languages in them.

I plan to add it to the AUR
I am still waiting for some dependencies.

Generally curious how many filetypes would have embedded languages like markdown can. In the frontend works, we can have html, css,and js/ts in a single file, but typically the formatter we use just handles it for us. For markdown specifically, prettier already can format the embedded language.

There's a lot of templating languages that have embedding (cshtml/razor/blazor, eelixir, erb, liquid). You can embed quite a few languages inside html other than javascript like py-script. I heard rust also supports "embedding" with macros. In lua/vim we have vimscript inside lua and the other way around. I think lisp can also be embedded in a lot of languages. It is hard to find a language where SQL cannot be "embedded" (in quotes because it is mostly embedded inside strings). There are probably more examples and I know a lot of these are relatively simple embeds or embeds inside strings, but it would be nice to support them if the implementation isn't overly complicated.

For markdown specifically, prettier already can format the embedded language.

Prettier only handles languages it can format on its own. If you for example have a python codeblock, it's not getting formatted.
I wrote cbfmt in a way that it is easy to add languages. You only need to add the treesitter parser and write one query. I am still thinking about how to do this though. Just adding ever possible parser in the binary would make it very big.

It would be nice if treesitter parsers were in some package manager. That way we wouldn't need to download them with nvim-treesitter and I believe this would be solved as well? As far as I know you only need the .dll/.so for a parser and queries?

I wrote a CLI tool that does exactly this https://github.com/lukas-reineke/cbfmt
It uses treesitter to find the codeblocks and formats them with any tool you want, it handles indentation, and it is parallel.
You should be able to just use it with formatter.nvim as well

Hi @lukas-reineke, could you provide a example for configuring cbfmt in formatter.nvim? Thank you!

Sure, would be just
{
  exe = "cbfmt",
  stdin = true,
}

Thanks @lukas-reineke . I try to run cbfmt directly in command line (cbfmt --config ./cbfmt.toml  test.md). But got error message:
[Fail]: test.md

[0/1] files are formatted correctly.

The input file is as follow:
```python
a = 1 +    100
```

Sorry, I accidentally removed the args above before posting :man_facepalming:
It should be
{
  exe = "cbfmt",
  args = {
    "--write",
    "--best-effort",
    "--stdin-filepath",
    util.escape_path(util.get_current_buffer_file_path())
  },
  stdin = true,
}

The error you get is because when you don't use --write it will just check if the file is formatted correctly.


Not able to load ViT-H_14

I was testing using the provided visualize_attention_map.ipynb
The ViT-B_16-224 loads fine but when I downloaded and was loading ViT-H_14, it gave me the following error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-36-0b02f0ab326a> in <module>
      2 config = CONFIGS["ViT-H_14"]
      3 model = VisionTransformer(config, num_classes=1000, zero_head=False, img_size=224, vis=True)
----> 4 model.load_from(np.load("imagenet21k_ViT-H_14.npz"))
      5 model.eval()

~/Documents/clones/ViT-pytorch/models/modeling.py in load_from(self, weights)
    287                 nn.init.zeros_(self.head.bias)
    288             else:
--> 289                 self.head.weight.copy_(np2th(weights["head/kernel"]).t())
    290                 self.head.bias.copy_(np2th(weights["head/bias"]).t())
    291

RuntimeError: The size of tensor a (1000) must match the size of tensor b (21843) at non-singleton dimension 0

What do you think might be the error?

Hello,
The reason for this error is that you are loading the imagenet21k weights instead of the fine-tuned version, so you need to set num_classes to 21843.
Note that the fine-tuned version of the ViT-H model is not listed as one of the available models, therefore if you need a model that is fine-tuned on the Imagenet 1k, you can use one of the models listed on the front page of this repository.


New Package: opensrp-ducks

It seems that all of the reducer modules implemented in the core client should be OpenSRP packages.
This is because chances are high that they need to be re-usable by other OpenSRP web implementations and not just the core client.

cc: @cKellyDesign

cc: @cKellyDesign

@moshthepitt When can we expect the opensrp npm scope to be created so that we can start publishing the packages.

@p-netm please create an account on NPM for OpenSRP.  Work with @jasonrogena or @ukanga if you need to do so.

@githengi @ndegwamartin ^^ just FYI.


Form page to add services.

Create the page in order to add new service : Lesson, assistance, hosts.


fix bug

When in thumbnail mode the script deletes the picture when done. I the if statement in line 175 mistakenly test for the wrong thing
    if url_in_description is True:

when instead it should be:
    if thumbnail_mode is True:

Thank you @Vraelot for pointing out this issue


Support the vscode-icons extension out of the box

I think that it really helps to see the file type immediately
https://github.com/robertohuertasm/vscode-icons

Thanks for reporting this issue. Please use the issue search next time to see if there is a duplicate already. In this case, the issue has already been reported.


remove typescript files npm ignore

the package is broken when npm installs it from another place (e.g. custom repository) If npm ignores typescript files

That will provide regression on issue https://github.com/troyanskiy/ng2-resource-rest/issues/25 which was fixed with that pull request https://github.com/troyanskiy/ng2-resource-rest/pull/26

I have published v 1.12.1
Please check change log.
To enable your way of mapping get params please set somewhere at app initialisation  ResourceGlobalConfig.getParamsMappingType = TGetParamsMappingType.Bracket

Thank you for your reply!
I think it's a little bit strange solution because when I install the package from any repository I have no main scripts at all.
I suggest using postinstall npm build and clean *.ts files by webpack or npm script.

That strange. We have Nexus repository and we did not have that kind of problem.
I will check that later.

Thanks!
http://take.ms/lovdn


refactor(firebase_database): firebase_database_platform_interface

Description

create separate firebase_database_platform_interface package
federate firebase_database plugin

Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See [Contributor Guide]).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[ ] The analyzer (melos run analyze) does not report any problems on my PR.
[x] I read and followed the [Flutter Style Guide].
[x] I signed the [CLA].
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

Yes, this is a breaking change.
[x] No, this is not a breaking change.

@googlebot I signed it!

@Salakar I have submitted this pull request which will create a separate firebase_platform_database_reference and federate the firebase_database.
But as per the comment https://github.com/FirebaseExtended/flutterfire/pull/1835#issuecomment-827475385, you also want to improve the integration tests.
Could you please guide or help to improve the integration tests?

@russellwheatley @Salakar

check all_plugins / analyze (pull_request) is failing because I have used the path dependency of firebase_database_platform_interface to federate the firebase_database plugin.

@deepak786 would you be willing to give your consent for co-authoring this? I've fixed my comments myself 😉

@googlebot I consent.

@googlebot I consent.

@lesnitsky is that you asked?

@deepak786 yes, thanks


Improve UnionArray logical_nulls tests

Which issue does this PR close?
Follow-up on #6303
Rationale for this change
As noted by @wiedld on https://github.com/apache/arrow-rs/pull/6303#discussion_r1783545074, the existing tests coverage is quite fragile relative to the target architecture and enabled CPU features, leaving most logic untested on ARM, for example.
What changes are included in this PR?
Instead of calling only the public Array::logical_nulls, tests of specific strategies also calls the correspondent private method directly. To do so, part of logical_nulls method, which builds the arguments to be passed to specific implementations, is made into a new private method UnionArray::fields_logical_nulls, and then used by both logical_nulls and the tests.
Are there any user-facing changes?
No, just tests

Just came back from FTO. I'll pick up this review tmrw. Thanks @gstvg !


DNN-10091: Update DotNetNuke.install.config.resources to support html5 video files

Hi @valadas , this only change in the refresh install, can you please also handle the upgrade scenario? thx

Hi @zyhfish I would be happy to, I will do some digging tomorrow, I am not sure where to implement that on upgrades. I will try to find that out tomorrow unless you already know and can point me in the right direction. Thanks.
Also, do you think it is a good idea to change allowed file types on upgrades. If for some reason I have a running website and I specifically set only .jpg are allowed, I might be surprised to see other extensions allowed after an upgrade without my knowledge and no warning? Let me know what you think.

Hi @valadas , you can check Website\Providers\DataProviders\SqlDataProvider\05.01.00.SqlDataProvider line 2852 to see how add extension during upgrade, the code change should be done in 9..2.0 data provider file.
we always have release note for what have been changed in every release, use should be know we add this in 9.2.0. if we don't catch upgrade scenario, then user may see the new toolbar in ck editor, but he can't upload video files, this will cause function broken, so we should take care of install/upgrade process in most of time. thx!

@zyhfish Makes a lot of sense, I will do that tomorrow. Thanks for the hint.

Thanks so much for your hard work, im reviewing the changes in CK editor now, I may add comment on that PR if thing need to be changed as well, thx again!

@zyhfish Ok, I got a little hickup with github, but I think the commits should be fine now. Let me know if there is any issue.

Hi @valadas , thx for the changes, i think it should be good enough. please resolve the conflicts, then i can merge into into main branch, thx!

It is done, sorry, I saw it before but I tought I needed write access to your repo to resolve it. I managed to do it, learned something new in the process. Let me know if there is anything else. And it is good to see pull requests getting processed, thanks for your help.

Changes looks good and merged, great thx for your contribution!


chore(lint): add misspell linter

this is super cool and it finds common misspells

/lgtm

let's see if I can approve my own PR
/apprve

/approve

/lgtm
/approve

We should decide if we should merge them or even use this CLI too #32

I am fine with using a tool for spell checking.. as far as it is opt-in and a human checks the diff first.


JS Warnings Over addDomListener()

Hello, I get a ton of JS warnings when using this library regarding the continued implementation of addDomListener(). Is there any chance of a pull that will remove this and use addEventListener() instead?

@tempranova Thank you for opening this issue. 🙏
Please check out these other resources that might be applicable:

Support Console - If you have a support contract.
Discord - chat with other developers
StackOverflow - use the google-maps tag
Google Maps Platform issue trackers

This is an automated message, feel free to ignore.

@tempranova Please take a moment to fill out this short survey. Thank you!
This is an automated message, feel free to ignore.


Fix issue where description gets returned as null to extension

Issue: Some websites, such as this mastodon page have no description and so the check endpoint was returning null to the extension for the description field.  When the user goes to save, they get the message "Error saving bookmark: Validation error: {"description":["This field may not be null."]}".
This error does not occur when metadata is loaded from the browser instead of the server.
Solution: When a description is not found an empty string will be returned.

Thanks for catching that! It's a bit strange though that the API would return an empty string if no content was found at all. And it would be inconsistent with other fields like the preview image for example, where it makes a lot of sense to return null if nothing was found.
Maybe the extension should make sure that it's not sending null to the API instead? That seems like a good idea in general as that might also cover other cases in the future where the description might end up having an unexpected value.

Is there a functional advantage to returning null from the API? Is there any situation that is making a decision based on it being null?  I considered changing the extension first but since all the other places that return description (browser metadata, existing bookmarks) return empty strings, this approach seemed the most consistent.
I considered changing the preview image too but I didn't have a good way to test downstream effects and it wasn't causing any errors on my side with the way it is currently written.

Is there a functional advantage to returning null from the API? Is there any situation that is making a decision based on it being null?

Not within linkding or the extension. But possibly in a 3rd party app or script that someone uses. Might be better to not change the behavior for that reason alone.
The issue here is the result of a change in the extension, so it seems better to deal with it there as well. Agree that it's inconsistent with what the extension does internally, but that seems like a lesser issue. Besides, the browser metadata logic could be changed to return null as well, and then the popup could deal with both cases similarly.

Closing this since we've decided to change the extension instead. See https://github.com/sissbruecker/linkding-extension/pull/86


Add new logo to page

What

The new logo will be added in place of the current title
Susanna produces the file or a series of files.
The files will be changed weekly from now until the event
The last file will be changing every time the page is loaded. The files could be static or they could be produced dynamically. Let's not prioritize this over key functionality.

Where

Main page template

Additionally let's set the logo as the page's favicon.
https://github.com/AvoinGLAM/h4og-dashboard/blob/production/frontend/public/index.html#L9

Until WP header can be removed, keep the header, remove our title. Later replace with logo.


zh: content/docs/examples/advanced gateways/egress gateway mtls origination

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rootsongjc
If they are not already assigned, you can assign the PR to them by writing /assign @rootsongjc in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content_zh/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Hi @loverto. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/retest

@loverto: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/joke

/reci

/status

/ok-to-test

/retest

@loverto Re-lint, please.

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: rootsongjc
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content_zh/OWNERS [rootsongjc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


How to set creation date on imported objects?

I have to migrate a bunch of mysql records into my Django application. The records already have a creation date and a 'last changed' date.
How do I import those dates into the history field?
I read about the custom history date field, but that is not what I am looking for. I don't want a separate field for the two dates I want to import. I want to have all the information in the history field.

@Borkason In this case, is necessary you create the history manually. You will need to create an instance of history with history_type=Created and set the history_date' with the creation date and another instance of history with history_type=Changed` and 'history_date' equals the last changed date.


am I doing this right?

Copied from https://github.com/lbryio/lbry/commit/f1e36823025199461836733e2a1b2e22db35596e, looks like it could work, but didn't test locally.

$ pylint --rcfile=setup.cfg torba
************* Module torba.basenetwork
torba/basenetwork.py:56:0: C0301: Line too long (111/110) (line-too-long)
torba/basenetwork.py:68:8: E1101: Instance of 'StratumClientProtocol' has no '_connected' member; maybe 'connected'? (no-member)
torba/basenetwork.py:73:8: W1201: Specify string format arguments as logging function parameters (logging-not-lazy)

Your code has been rated at 9.97/10
Lol
I used the web gui, so no formating was performed.


Heading glitches in small resolutions

first to thank you because Bumbag is the best UI library for React out there at the moment. It is wonderful !
Describe the bug
at small resolutions the Heading component fails and styles are lost
To Reproduce

Install bumbag for nextjs

Inside the index page import the Heading component

Activate toggle device toolbar from the browser

Expected behavior
The component should keep its style
Screenshots

Something curious in some ios devices the Heading component does work

My code:
// _document.js
import Document, { Head, Html, Main, NextScript } from "next/document";
import { extractCritical } from "bumbag-server";
import { InitializeColorMode } from "bumbag";
export default class MyDocument extends Document {
  static async getInitialProps(ctx) {
    const initialProps = await Document.getInitialProps(ctx);
    const styles = extractCritical(initialProps.html);
    return {
      ...initialProps,
      styles: (
        <>
          {initialProps.styles}
          <style
            data-emotion-css={styles.ids.join(" ")}
            dangerouslySetInnerHTML={{ __html: styles.css }}
          />
        </>
      ),
    };
  }
  render() {
    return (
      <Html>
        <Head />
        <body>
          <InitializeColorMode />
          <Main />
          <NextScript />
        </body>
      </Html>
    );
  }
}

// _app.js
import NextApp from "next/app";
import { Provider as BumbagProvider } from "bumbag";
export default class App extends NextApp {
  render() {
    const { Component, pageProps } = this.props;
    return (
      <BumbagProvider isSSR>
        <Component {...pageProps} />
      </BumbagProvider>
    );
  }
}

//Index
import { Box, Heading } from "bumbag";

const Home = () => {
  return (
    <Box>
      <Heading use="h1">Heading fails</Heading>
    </Box>
  );
};

export default Home;

Testing on my physical Redmi phone I noticed the Heading component also has problems
Thank you!

can you replicate this please? @jxom

FWIW, I can replicate this also (see https://subscribe.flightpenguin.com)