Dataframe index is not sorted after .set_index()

From the docs+source it appears that the output of ddf.set_index() should be sorted by index, but this doesn't seem to be the case.
import dask, dask.dataframe as dd, distributed

In [1]: c = distributed.Client('localhost:8786')
In [2]: ddf = dd.read_csv('gs://<bucket>/2018-01-*.csv.gz', compression='gzip').set_index('uid').compute()
In [3]: ddf.index.is_monotonic
Out[3]: False
In [4]: ddf.head().index
Out[4]:
Index(['00004cd2-8a03-4dc7-85a9-f32ded88d6f6',
       '00004cd2-8a03-4dc7-85a9-f32ded88d6f6',
       '00009051-c962-460b-8090-c1c9d45df180',
       '00009051-c962-460b-8090-c1c9d45df180',
       '00009051-c962-460b-8090-c1c9d45df180'],
      dtype='object', name='uid')

The inputs are randomly ordered, and the output appears to be mostly-but-not-quite sorted; maybe sorted within each partition but not reassembled correctly?

I suspect that it is the reverse, data has been sorted between partitions
but not within.  My guess is that this does the minimum work to ensure that
partitions/divisions are consistent.  Sorting the index on set_index seems
sensible to me.  Any interest in submitting a PR?
If memory serves this was discussed before, I'm a little surprised to see
that it was not done.  It may be that there were decent counter-arguments
at the time.
On Thu, Apr 5, 2018 at 12:29 PM, Brett Naul<EMAIL_ADDRESS>wrote:

From the docs+source it appears that the output of ddf.set_index() should
be sorted by index, but this doesn't seem to be the case.
import dask, dask.dataframe as dd, distributed
In [1]: c = distributed.Client('localhost:8786')
In [2]: ddf = dd.read_csv('gs:///2018-01-*.csv.gz', compression='gzip').set_index('uid').compute()
In [3]: ddf.index.is_monotonic
Out[3]: False
In [4]: ddf.head().index
Out[4]:
Index(['00004cd2-8a03-4dc7-85a9-f32ded88d6f6',
'00004cd2-8a03-4dc7-85a9-f32ded88d6f6',
'00009051-c962-460b-8090-c1c9d45df180',
'00009051-c962-460b-8090-c1c9d45df180',
'00009051-c962-460b-8090-c1c9d45df180'],
dtype='object', name='uid')
The inputs are randomly ordered, and the output appears to be
mostly-but-not-quite sorted; maybe sorted within each partition but not
reassembled correctly?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3376, or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszAE7AjPYYrmeARuD0VCBBsCiuYXjks5tlkaHgaJpZM4TIxlW
.

Interesting; do you think things like .groupby().apply() would be affected by this level of non-sortedness, or will everything behave OK as long as the partitions/divisions are consistent? The fact that the index isn't sorted within partitions might not be a problem per se, I was mostly just surprised since I saw several references to .set_index performing a sort.

Most pandas operations don't assume a sorted index, some do though.  It
could be that operations like loc would misbehave?
Dask dataframe logic only assumes that divisions are accurate and that
pandas operations work.  Sortedness probably isn't required.
The right person to ask about correct behavior here is probably
@TomAugspurger
On Thu, Apr 5, 2018 at 1:11 PM, Brett Naul<EMAIL_ADDRESS>wrote:

Interesting; do you think things like .groupby().apply() would be
affected by this level of non-sortedness, or will everything behave OK as
long as the partitions/divisions are consistent? The fact that the index
isn't sorted within partitions might not be a problem per se, I was mostly
just surprised since I saw several references to .set_index performing a
sort.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3376#issuecomment-379008607, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszPAFnHPA0k7cban_1-C2GsxHDOq9ks5tllAsgaJpZM4TIxlW
.

I am interested; will try to take a look within the next few days

@mrocklin I poked around a bit more: I have only been able to reproduce this behavior w/ the distributed scheduler and lot of data, so unfortunately I don't have a simple test case to share (nor can I use the single-threaded scheduler for debugging).
It seems like sorting is not the issue per se, but rather the shuffle/partition itself:
ddf = dd.read_csv('gs://<bucket>/2018-01-*.csv.gz', compression='gzip').set_index('uid').persist()

# found a partition where one order violation occurred
In [90]: ddf.get_partition(255)
Out[90]:
Dask DataFrame Structure:
                                          lat      lng
npartitions=1
135c2fc8-8861-4dfd-9663-4b79438d311f  float64  float64
137266de-0abd-473c-9915-1f1330d58d71      ...      ...

In [92]: ddf.map_partitions(lambda df: df.index.is_monotonic).compute().all()
Out [93]:
True

The divisions themselves are in sorted order, and each partition individually is sorted, but a few (in this case 6, out of a 4M row dataframe) just end up in the wrong place. Any ideas how this might come to be?

Yep this is 0c443fd49cd63ce23f803706c7f9762f0b8bb165. I think I could come up with a reproducible example if it would help, but like I said it seems like a rare problem and I can't reproduce it on anything smaller than millions of rows. I have managed to get the same behavior w/ a subset of columns: now I have a few thousand files that look like
i,uid
0,002862bc-67d5-43a4-bbb7-da659a987f0c
1,002862bc-67d5-43a4-bbb7-da659a987f0c
2,002862bc-67d5-43a4-bbb7-da659a987f0c
3,002862bc-67d5-43a4-bbb7-da659a987f0c
4,002862bc-67d5-43a4-bbb7-da659a987f0c
5,03b076de-9038-48d3-9609-178690ca808e
6,03b076de-9038-48d3-9609-178690ca808e
7,03b076de-9038-48d3-9609-178690ca808e
8,03b076de-9038-48d3-9609-178690ca808e
9,03b076de-9038-48d3-9609-178690ca808e

Loading these, setting the index, and computing gives the above behavior.

The following runs in about 5 minutes on my local machine:
import numpy as np, pandas as pd, dask.dataframe as dd, distributed, uuid

c = distributed.Client('localhost:8786')

ids = [uuid.uuid4() for i in range(100_000)]
df = pd.DataFrame({'uid': np.concatenate([np.repeat(uid, np.random.randint(1, 750)) for uid in ids]),
                   'x': 0})
dd.from_pandas(df, npartitions=4000).to_csv('/tmp/test_*.csv', index=False)
del df

ddf = dd.read_csv('/tmp/test_*.csv').set_index('uid').persist()
print(ddf.index.compute().is_monotonic)  # False

For me the very first partition was incorrect:
In [35]: ddf.get_partition(0)
Out[35]:
Dask DataFrame Structure:
                                          x
npartitions=1
00001f4f-7f25-43d0-a939-cc485a359e7e  int64
00143d26-016c-427a-84ff-8d1fca4bfa08    ...
Dask Name: get-partition, 4001 tasks

In [36]: ddf.get_partition(0).tail()
Out[36]:
                                      x
uid
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0

Hrm, I'm curious, do you need read_csv and distributed to reproduce this?
On Mon, Apr 23, 2018 at 1:53 PM, Brett Naul<EMAIL_ADDRESS>wrote:

The following runs in about 5 minutes on my local machine:
import numpy as np, pandas as pd, dask.dataframe as dd, distributed, uuid
c = distributed.Client('localhost:8786')
ids = [uuid.uuid4() for i in range(100_000)]
df = pd.DataFrame({'uid': np.concatenate([np.repeat(uid, np.random.randint(1, 750)) for uid in ids]),
'x': 0})
dd.from_pandas(df, npartitions=4000).to_csv('/tmp/test_*.csv', index=False)
del df
ddf = dd.read_csv('/tmp/test_*.csv').set_index('uid').persist()
print(ddf.index.compute().is_monotonic)  # False
For me the very first partition was incorrect:
In [35]: ddf.get_partition(0)
Out[35]:
Dask DataFrame Structure:
x
npartitions=1
00001f4f-7f25-43d0-a939-cc485a359e7e  int64
00143d26-016c-427a-84ff-8d1fca4bfa08    ...
Dask Name: get-partition, 4001 tasks
In [36]: ddf.get_partition(0).tail()
Out[36]:
x
uid
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
f5f3f922-b22c-4537-90fe-f8a311d26f14  0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3376#issuecomment-383664841, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszA0Wnbk00N25ZXoT50hFVHn0SX8Pks5trhT8gaJpZM4TIxlW
.

Yep, shuffle='tasks' gives the same incorrect behavior. I tried decreasing the size of the dataframe but with fewer partitions it seems to always end up sorted (4000 breaks, 2000 is OK). Not sure what to make of that 🙃

Not sure what changed but when I run the above example now the index is sorted. Guess I'll close...?


Update compass config

Adds sourcemaps and nested CSS format for debugging. Also sets config options for more consistent builds across environments.
Because there was a change to config.rb compass does a 'clean' before compiling which is the reason for all the CSS and sprite changes in this commit.
Compass' default environment is :development which we've been treating as :production until now. So compass compile now generates CSS in an expanded format and also generates sourcemaps. For pushing back upstream we should use compass compile -e production.
/cc @jonom

So excited for source maps! I'm not a huge fan of the development/production approach here though - is it harmful to leave the source maps on all the time? People can access the scss files in a browser either way so I don't think it poses a security risk. I'm not sure there's much advantage doing nested output for dev versions either, most people would view the CSS in dev tools where it's reformatted anyway wouldn't they? And if they have to ultimately edit an SCSS file to change things I don't think it really matters if the CSS output file is more readable. With this approach potential contributors would have to set up the project and re-compile the SCSS just to trace a style to the source, think that barrier will deter a lot of community members from investigating and fixing a CSS problem.
In short I'd much rather leave source maps turned on all the time and always produce compact output so we don't create a lot of extra compiling work.

I've added sourcemaps to .gitignore so they wont be in the repo.
Yeah that makes sense - I'll update it so we're always generating compact CSS.

Update so we're always generating compact CSS and always generating sourcemaps

Why not include the sourcemaps in the repo?

There's already a comment in the config which says 'Disable comments in the output. We want admin comments to be verbose'. Sourcemaps display comments, so I've excluded them. I don't think it's security issue, so they might be OK to include, anyone else want to weigh in here? They're not a performance issue as they're only requested if devtools is open.

It's an opensource project though - so you can see the comments if you really want to...

I think that line means that we want lots of comments in the source file, but that bloats the file, so we won't include them in the output that gets sent to people's browsers. Source maps and the target SCSS files are only loaded by the browser if you have dev tools open so should be all good I think :)

They're not a performance issue as they're only requested if devtools is open.

Oh you already said that heh

OK - sorry for the mucking around. I've included them :smile:

Nice! Thanks. Looks great to me.

I approve of sourcemaps, but don't tend to need them myself. But whatever helps other devs track things down is a gain. :+1:

Does anyone have a feeling on whether unix_newlines as standard is appropriate for css output? Shouldn't text transmitted by HTTP use <some standard> newline?

I'm not sure it matters?
From: http://sass-lang.com/documentation/file.SASS_REFERENCE.html#unix_newlines-option
:unix_newlines
If true, use Unix-style newlines when writing files. Only has meaning on Windows, and only when Sass is writing the files (in Rack, Rails, or Merb, when using Sass::Plugin directly, or when using the command-line executable).

Yeah I added it to increase the consistency of output across environments. I read the docs @adrexia posted when I added the option, and read it to mean unix newlines are used by default, as it's only used on Windows.


Make deriveTargetSwiftSDK public

Used by https://github.com/swiftlang/sourcekit-lsp/pull/1643
Motivation:
Allows us to achieve parity between sourcekit-lsp's configuration options and SwiftPM's --triple/--swift-sdk.
Modifications:
Make SwiftSDK.deriveTargetSwiftSDK public
Result:
See https://github.com/swiftlang/sourcekit-lsp/pull/1643

@swift-ci test

@MaxDesiatov think we're ready to merge this!


ItemViewBinder can't be recycled

There is a VideoView in one of the ItemViewBinder and it was still playing when Activity destroyed.
I try to release the VideoView but failed  because onViewDetachedFromWindow and onViewRecycled was not called

Impossible. You need to provide evidence and sample project.
In addition, you can refer to the official sample:
https://github.com/drakeet/MultiType/blob/3eaf09c978fb335f9b5ecded1e8b875a513a23aa/sample/src/main/java/me/drakeet/multitype/sample/moreApis/MoreApisPlayground.java#L97-L146


[XLA][AMDGPU] enable AMDGPU target

Enable AMDGPU target in XLA GPU build

@jlebar This is the very first PR related to enabling AMDGPU target for XLA. I tried to make this PR minimal and doesn't depend on the bigger PR #20277 which is still under review.

@qlzh727 would you be willing to merge this for us?

The build error in XLA test target could be resulted from recent TableGen separation in upstream LLVM between AMDGPU and legacy R600 architecture. I'll update the PR.

@jlebar / @d0k I've revised the PR to address build issues encountered earlier. Could you review once again? Thanks.
Moving forward I think it'd be good to open source the script to generate llvm.autogenerated.BUILD.

@d0k should review this, as I'm not familiar with the correct output here.

@jlebar, at least one googler need to approve the PR. Can u add your review/approval?

We have an internal change to do this which should subsume this PR.  @d0k will comment here once that's submitted, then we should just be able to close this as a nop.

ee9a16b2032c8cb96180b0e81fbae3076b54a883 should do it. I hope I got everything right.


Implement profile page with editable fields

Implemented the profile page with functionality to display and edit user information.
This allows users to view and update their profile details directly from the interface.

Looks good to me


Bug using flag "--with-rand-seed" for non x86 architectures

The build.info file found under rands/seeding/build.info has several condition relating to the type of the target configuration.
Here is the code from that file:
$COMMON=rand_unix.c rand_win.c rand_tsc.c
IF[{- $config{target} =~ /vxworks/i -}]
  $COMMON=$COMMON rand_vxworks.c
ENDIF
IF[{- $config{target} =~ /vms/i -}]
  $COMMON=$COMMON rand_vms.c
ENDIF
IF[{- !$disabled{asm} && $config{target} =~ '.*aarch64' -}]
  $COMMON=$COMMON rand_cpu_arm64.c
ELSE
  $COMMON=$COMMON rand_cpu_x86.c
ENDIF

SOURCE[../../../libdefault.a]=$COMMON

I believe that due to oversight, the conditions don't take into consideration the fact that the target architecture configured to use openssl with the requested flag might differ from all the options provided by these conditions.
In addition, since the default behavior for not following any of the conditions provided in the file is using the file "rand_cpu_x86.c", builds for architectures (such as PPC, MIPS, etc) may fail due to the usage of an incompatible implementation for the target architecture.
A quick fix for this issue can be to change the "ELSE" in the file into a condition, as described in the following code block:
IF[{- $config(target) =~ '.*x86.*' -}]

This way, the usage of "rand_cpu_x86.c" will be guaranteed to only be used for x86 architectures (such as IA32, and AMD64) .

I believe that due to oversight, the conditions don't take into consideration the fact that the target architecture configured to use openssl with the requested flag might differ from all the options provided by these conditions.

I'm somehow missing the point of the bug you are trying to report here. Can you give a concrete example of how we can go wrong here?

It's not ideal and ought to be addressed but I'm not seeing a bug here.
You have to explicitly configure with --with-rand-seed=rdcpu when building for a platform where that isn't supported.
I'd suggest choosing another seed source that is supported for that target.
If you've got a specific target where you want this option supported, feel free to submit a pull request.

In INSTALL.md, it is explicitly mentioned:
rdcpu
Use the RDSEED or RDRAND command on x86 or RNDRRS command on aarch64 if provided by the CPU.
I interpret that meaning you cannot use rdcpu on other architectures.
Please feel free to submit a pull request with any documentation improvements you see fit.

The problem is basically that the default behavior uses the x86 architecture file, while not supporting any other that may still need to use the flag.

Only x86 and aarch64 currently support this flag. To support additional architectures, changes similar to those in #15361 would need to be made for that architecture.


flask import not  found

Thanks for the report. Would you please check if the right virtual environment is being used?
The way to do that would be to:

If you have not selected a virtual environment, the status bar will show the global python interpreter.
Click on version number and choose the right environment from the command palette
If there is no environment offered, you will need to create an environment and then select that as the default environment
When you select the right virtual environment Pylance find the modules installed in that environment.

Please respond and let us know if that worked.

Closing inactive issue, please reopen if this is still an issue


VS Code isn't outputting the output of the pre-push git hook script

Vs Code isn't outputting the output of the pre-push git hook script script nor the output of any triggered batch-files (*.bat) triggerd by it.
I initially thought this was an issue in Git for Windows, so started an issue message over there (https://github.com/git-for-windows/git/issues/3419), but it works when using git push in a commandline and when searching on google more people were having the exact same issue with the pre-push hook not outputting in VS Code's output panel.
An example of other people having this issue with pre-push:
https://developercommunity.visualstudio.com/t/git-integration-does-not-show-local-hooks-output/153462
According to this thread (https://github.com/microsoft/vscode/issues/59787) it would've be fixed in VS Code years ago, but it doesn't work in the latest VS Code.
Some other hooks DO output its output to VS Codes git panel though. The hooks pre-commit, prepare-commit and commit-msg perfectly show their output in the output panel and also a batch-file (*.bat) which was triggered by the commit-msg hook here has its output shown in the VS Code git output panel.
So I would expect the pre-push output to be shown there as well, the same with the batch-file triggered from there.

VS Code Version: 1.60.0
OS Version: Windows 10

Steps to Reproduce:

Create a pre-push script inside .git/hooks which echos a string and triggers a batch file

#!/bin/sh

echo '----------------- PRE PUSH ------------------'

cmd.exe /c "test.bat"

exit 0

Create a test.bat batch script inside the project folder which echos a string and creates a folder

@echo --------------- BATCH FILE EXECUTING ----------------

mkdir test-created-by-push

Commit the batch file to git
Open the Output panel on Git
Use VS Code's build in git push
Look at the Output > Git panel and see there's no indication of the hook being fired; no echo from the hook and no echo from the batch file. The folder is created though, so the script was executed while pushing.

Example of pre-push hook that does NOT show the output of the hook nor the output of the bat-file

Example of hooks that DO show the output of the hook as well as the output of the triggered batch-file (*.bat)

Example of the pre-push to indicate it IS working from 'just a' commandline:

Can confirm this when using husky

same problem for me using husky

Have the same issue, by using hooks via https://github.com/evilmartians/lefthook (but it just creates a regular Git hooks).
Also I am using VSCode remote container (git is running inside container, Debian based), host machine is also Debian based Linux. So it looks like not related to Windows.

Is there something we can do to poke this or see its priority? I'm running into this as well and seeing nothing on it since September is discouraging.

The output of the git client-side hooks is being written to stdout and by default only stderr is being logged in the "Git output" channel. The Insiders release on Monday will have a new setting (git.commandsToLog) that one can use to specify an array of git commands that will have their stdout (including the output of the client-side hooks if there are any configured) logged to the "Git output" channel.
Try out the new setting and update the issue if you see any problems with it.

@lszomoru Thanks! Great to 'hear' it's coming!
One question tho; you write only the stderr is currently being logged, as being the reason the hook is not outputed in vs code. But some of the hooks' outputs ARE shown in VS Code (see opening post).  How come these hooks ARE shown while not being errors?

@bencresty, that is a fair question. Looking at the code, I am fairly confident that before my change, we have only logged the contents of stderr. So my only hypothesis at the moment is that somehow that content ends up in stderr and that is why it gets written into the Git output channel.

Steps to verify:

Create a new folder (ex: hooks-test) and run git init in the folder
Open hooks-test\.git\hook and create a new file called pre-push
Add the following to the file#!/bin/sh
echo HELLO WORLD!

Open the hooks-test folder in VS Code Insiders
Open the Git output channel using the Git: Show Git Output... command
Make a file change, commit, and push the change

Confirm that you do not see "HELLO WORLD!" in the Git output channel

Open settings and find the git.commandsToLog setting. Add push to it.
Make a file change, commit, and push the change

Confirm that you see "HELLO WORLD!" in the Git output channel


AzureFileCopyV4: Storage account not found but exists in portal and storage explorer

Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: AzureFileCopyV4
Environment
Server: Azure Pipelines
Build ID: 60
Build number: 3900
Agent: Microsoft Hosted windows-latest
Issue Description
The task reports that the storage account cannot be found even though the storage account clearly exists in the Azure Portal as well as the Azure Storage Explorer. The storage account was deployed several hours ago.
Error message:
Storage account: [storageAccountName] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.

Things I've tried:

Rerun failed job: Failed
Upload via Storage Explorer: Success
Local upload with azcopy v10 authenticated with my account: Success
Local upload with azcopy v10 authenticated with same service principal as pipeline: Success

Local command:
azcopy copy "C:\form-schemas\*" "https://[storageAccountName].blob.core.windows.net/formschemas" --recursive=true --overwrite=false

Task description
- task: AzureFileCopy@4
  displayName: 'Upload form-schemas'
  inputs:
    SourcePath: '$(Agent.BuildDirectory)/form-presenter-output/formSchemas/*'
    azureSubscription: ${{parameters.azureConnectionName}}
    Destination: AzureBlob
    storage: [storageAccountName]
    ContainerName: formschemas
    AdditionalArgumentsForBlobCopy: '--recursive=true --overwrite=false'

Task logs
See attached zip file.
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
2021-03-24T11:42:31.8582403Z ##[debug](ARM)Storage account: [storageAccountName] not found
...
2021-03-24T11:42:34.0770501Z ##[debug]Caught exception from task script.
2021-03-24T11:42:34.0804375Z ##[debug]Error record:
2021-03-24T11:42:34.1352952Z ##[debug]Storage account: [storageAccountName] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2021-03-24T11:42:34.1365111Z ##[debug]At D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.184.1\AzureUtilityAz1.0.ps1:22 char:13
2021-03-24T11:42:34.1375852Z ##[debug]+             Throw (Get-VstsLocString -Key "AFC_StorageAccountNotFound ...
2021-03-24T11:42:34.1387386Z ##[debug]+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-03-24T11:42:34.1399293Z ##[debug]    + CategoryInfo          : OperationStopped: (Storage account...ager type only.:String) [], RuntimeException
2021-03-24T11:42:34.1410692Z ##[debug]    + FullyQualifiedErrorId : Storage account: [storageAccountName] not found. The selected service connection 'Service Pr    incipal' supports storage accounts of Azure Resource Manager type only.
2021-03-24T11:42:34.1421254Z ##[debug]
2021-03-24T11:42:34.1439590Z ##[debug]Script stack trace:
2021-03-24T11:42:34.1473298Z ##[debug]at Get-AzureStorageAccountResourceGroupName, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.184.1\AzureUtilityAz1.0.ps1: line 22
2021-03-24T11:42:34.1484915Z ##[debug]at Get-AzureStorageKeyFromARM, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.184.1\AzureUtilityRest.ps1: line 11
2021-03-24T11:42:34.1495985Z ##[debug]at Get-StorageKey, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.184.1\Utility.ps1: line 70
2021-03-24T11:42:34.1506789Z ##[debug]at <ScriptBlock>, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.184.1\AzureFileCopy.ps1: line 104
2021-03-24T11:42:34.1518102Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-03-24T11:42:34.1529118Z ##[debug]at <ScriptBlock>, <No file>: line 22
2021-03-24T11:42:34.1540556Z ##[debug]at <ScriptBlock>, <No file>: line 18
2021-03-24T11:42:34.1551517Z ##[debug]at <ScriptBlock>, <No file>: line 1
2021-03-24T11:42:34.1570469Z ##[debug]Exception:
2021-03-24T11:42:34.1588528Z ##[debug]System.Management.Automation.RuntimeException: Storage account: [storageAccountName] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.
2021-03-24T11:42:34.1772788Z ##[error]Storage account: [storageAccountName] not found. The selected service connection 'Service Principal' supports storage accounts of Azure Resource Manager type only.

log.zip

is it classic storage account?

The account is a StorageV2 kind. Below is the complete ARM template for its deployment:
{
  "type": "Microsoft.Storage/storageAccounts",
  "apiVersion": "2019-06-01",
  "name": "[variables('storageAccountName')]",
  "location": "[parameters('location')]",
  "sku": {
    "name": "Standard_LRS"
  },
  "kind": "StorageV2",
  "properties": {
    "accessTier": "Hot",
    "minimumTlsVersion": "TLS1_2",
    "supportsHttpsTrafficOnly": true,
    "allowBlobPublicAccess": false
  },
  "resources": [
    {
      "type": "blobServices",
      "apiVersion": "2019-06-01",
      "name": "default",
      "dependsOn": ["[variables('storageAccountName')]"],
      "properties": {
        "cors": {
          "corsRules": [
            {
              "allowedHeaders": ["*"],
              "allowedOrigins": ["*"],
              "maxAgeInSeconds": 0,
              "allowedMethods": ["GET", "OPTIONS", "HEAD"],
              "exposedHeaders": ["*"]
            }
          ]
        },
        "isVersioningEnabled": true,
        "deleteRetentionPolicy": {
          "enabled": true,
          "days": 90
        }
      }
    },
    {
      "type": "blobServices/containers",
      "apiVersion": "2019-06-01",
      "name": "default/formschemas",
      "dependsOn": ["[variables('storageAccountName')]"],
      "properties": {
        "publicAccess": "None"
      }
    }
  ]
}

After failing 8 times in a row it suddenly worked once then failed again twice. All with the exact same error. The Azure Resources have not changed between runs.

Its failing at this command
Get-AzureRmStorageAccount -ErrorAction Stop | Where-Object { $_.StorageAccountName -eq $storageAccountName }
can you run this locally/AzurePowershell task using same service principal?

Hi @AmrutaKawade I've run the command locally and it works fine (using Az module):
PS C:\> Get-AzStorageAccount -ErrorAction Stop | Where-Object { $_.StorageAccountName -eq $storageAccountName }

StorageAccountName ResourceGroupName    PrimaryLocation SkuName      Kind      AccessTier CreationTime        ProvisioningState EnableHttpsTrafficOnly LargeFileSh
                                                                                ares
------------------ -----------------    --------------- -------      ----      ---------- ------------        ----------------- ---------------------- -----------
[storageAccountName]   [resourceGroup] westeurope      Standard_LRS StorageV2 Hot        25.03.2021 15:40:16 Succeeded         True

Since yesterday evening our builds are working again as well without any changes. All I did was re-queue it so I don't actually have a failing environment to test with at the moment. I will remove it and try again to reproduce the error.
Could there be a delay in some registry somewhere? I was able to see the storage account in the azure portal as well as Azure Storage Explorer client, but could there be another registry used by the PowerShell modules or something like that which has a delay on updates?

@rudfoss I am not sure about powershell modules but some ARM rest APIs read data from cache and even if you see resources immediately on portal, API will take 30-40 min to return those resources.

@AmrutaKawade Understood. I expect this is the problem then. We've usually always been able to successfully run the job after about an hour or so. Is there anything we can do to mitigate this? We are deploying new environments from scratch with every PR in  order to test both clean installs and upgrade scenarios so it would be great if we didn't have to wait for an hour every time a new PR environment needs to be provisioned.

@AmrutaKawade I've deployed environments a couple of times now and the error has not resurfaced. Not sure how I can go about reproducing it now. As soon as I see it again I'll try running Get-AzStorageAccount in context and report back.

I'm pretty sure I hit the same thing today. Strangely, on first deployment everything worked fine. But then every subsequent deployment over the next 2.5 hours failed with the same error message as OP had. I took lunch, came back an hour later and it's working again.


Data discovery API

Enable scientists to easily find curated datasets, and annotate and share personal datasets.

Closed with https://github.com/astronomy-commons/hipscat/pull/103


feat: new SM v2 API release

BREAKING CHANGE: new v2 API.

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Provide guidelines for adding new tests under e2e

OpenEBS developer documentation should include how to extend the e2e framework for OpenEBS

https://github.com/openebs/openebs/blob/master/contribute/CONTRIBUTING-TO-CI-AND-E2E.md


Does anyone have pretrained inception-resnet-v2 model?

I didn't find it on mxnet model zoo :sob:
Thanks a lot! :laughing:

the same problem as you

@iFighting Do u play Kaggle?

@ysh329 yep,i play

@iFighting I emailed u.

@ysh329 @iFighting.  I am also looking for pretrained inception-resnet-v2 model. Where can I download one? Or could you please email me one copy? Thank you.

@TerryLovesLife Excuse, we don not have pre-trained inception-resnet-v2 models, either. :disappointed:

I am also looking for inception-v4 and inception-resnet-v2, does anyone have them?


ValueError: The metric [context_precision] that that is used requires the following additional columns ['reference'] to be present in the dataset.

[ ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
A clear and concise description of what the bug is.
Ragas version:0.1.18
Python version:3.10.9
Code to Reproduce
from agent import *
from datasets import Dataset
data = {
"question": ["How does Qingdao strengthen the coordinated governance of land, coast, and sea in its '14th Five-Year' ecological and environmental protection plan?"],
"answer": ["In Qingdao's '14th Five-Year' ecological and environmental protection plan, the following measures are taken to strengthen the coordinated governance of land, coast, and sea:\n\n1. Establish a land-sea coordination mechanism: Strengthen the coordinated governance of coastal, river basin, and sea areas, and promote the convergence of policies and standards across regions.\n\n2. Enhance pollution prevention and ecological protection: Coordinate pollution control in river basins and sea areas to ensure the connection and interaction of ecological protection and restoration responsibilities.\n\n3. Establish a water quality monitoring mechanism: Establish a response mechanism for coastal water quality and river inflows to ensure the organic integration of water quality monitoring and pollution prevention efforts.\n\nThese measures aim to achieve continuous improvement in the marine environment, build a beautiful ocean, and enhance the public's sense of gain and happiness when accessing the sea.\n\nSource: Qingdao Municipal People's Government Notice on Issuing the '14th Five-Year' Ecological and Environmental Protection Plan of Qingdao."],
"contexts": [
[
"With the protection and construction of beautiful bays as the guiding principle, we systematically plan land, coast, and sea governance tasks to promote continuous improvement in the ecological environment of coastal waters, strictly abide by the bottom line for marine biological ecology to rest and recuperate, prevent and defuse marine environmental risks, build a 'blue sea, clear sky, clean beaches' beautiful ocean, and enhance the public's sense of gain and happiness when accessing the sea.\n(I) Strengthen the coordinated governance of land, coast, and sea.\n1.  Strengthen the comprehensive governance of coastal, river basin, and sea areas in a coordinated manner, enhance the convergence of ecological and environmental goals, policies, standards, and systems in coastal districts (cities), rivers flowing into the sea, and coastal waters, strengthen the convergence, coordination, and unified supervision of pollution prevention and ecological protection and restoration responsibilities in regions, river basins, and sea areas, promote the coordinated governance of river basins and sea areas, and establish a response mechanism for coastal water quality - river inflow sections - land-based pollution sources.",
"Rivers and animal migration corridors, constructing an ecological spatial pattern of 'one bay with two wings, three mountains and one plain, one axis with multiple corridors'. Implement classified control of land and space, and establish an ecological and environmental zoning control system centered on ecological protection red lines, environmental quality baselines, resource utilization upper limits, and ecological and environmental access lists. Improve the management system of nature reserves with 'classified setting, hierarchical management, and zoning control', integrate and optimize various types of nature reserves, and protect biodiversity.\nPromote the construction of beautiful bays. Implement the Jiaozhou Bay Protection Regulations, strengthen the permanent protection of coastlines, strictly prohibit land reclamation, implement wetland protection, ecological restoration, marine biological resource restoration and enhancement projects, raise the threshold for industrial access in the bay area, and strengthen the assessment of water environment and ecological conditions. Establish a linkage mechanism for pollution prevention and control in river basins, rivers, and coastal waters, and implement pollution prevention and control measures in key sea areas from land sources.",
"Implement strict control over beach reclamation and sea reclamation, strictly control development and utilization activities in strictly protected areas, restricted development areas, and sea areas with weak ecological vulnerability and self-purification capabilities such as Jiaozhou Bay, Dingzi Bay, Tangdao Bay, Lingshan Bay, Longwan, and other areas. Strengthen maritime law enforcement, establish a normalized working mechanism for joint maritime law enforcement, strengthen the supervision and inspection of sea use. Expand the scope of mortgage loans for sea area use rights, and explore the development of mortgage loan business for island use rights. Promote the integration of port resources and operating resources, comprehensively clean up various small and scattered ports with close locations, overlapping functions, inefficient utilization, and serious pollution, and encourage the conversion of some construction sea space into marine ecological space.\n(II) Strengthen marine ecological protection and restoration\nImplement joint prevention and control of land-sea environmental pollution, strengthen the supervision and management of direct discharge outlets into the sea, and strengthen the environmental protection of river basins and coastal waters."
],
],
"ground_truths": [
["The methods Qingdao adopts in its '14th Five-Year' ecological and environmental protection plan to strengthen the coordinated governance of land, coast, and sea include: establishing a land-sea coordination mechanism, strengthening the comprehensive and coordinated governance of coastal, river basin, and sea areas, enhancing the convergence of ecological and environmental goals, policies, standards, and systems in coastal districts (cities), rivers flowing into the sea, and coastal waters, strengthening the convergence, coordination, and unified supervision of pollution prevention and ecological protection and restoration responsibilities in regions, river basins, and sea areas, promoting the coordinated governance of river basins and sea areas, and establishing a response mechanism for coastal water quality - river inflow sections - land-based pollution sources."]
]
}
dataset = Dataset.from_dict(data)
使用RAGAs进行评估
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_recall,
context_precision,
)
result = evaluate(
dataset,
metrics=[
context_precision,
context_recall,
faithfulness,
answer_relevancy,
],
)
print(result)
Error trace
Traceback (most recent call last):
File "D:.study\myProject\hh.py", line 32, in
result = evaluate(
File "F:\Application2022\Anaconda\an\lib\site-packages\ragas_analytics.py", line 129, in wrapper
result = func(*args, **kwargs)
File "F:\Application2022\Anaconda\an\lib\site-packages\ragas\evaluation.py", line 177, in evaluate
validate_required_columns(dataset, metrics)
File "F:\Application2022\Anaconda\an\lib\site-packages\ragas\validation.py", line 62, in validate_required_columns
raise ValueError(
ValueError: The metric [context_precision] that that is used requires the following additional columns ['reference'] to be present in the dataset.
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

Please refer to my answer in #1299


Union Update / Create input object schemas with Unchecked

Description
Resolves #23

Looks good to me. But we just need one more change for Upsert and that would cover everything. I'll push it to this PR now

Sweet!


As an HR advisor I want to produce a report on the salary of all employees so that I can support financial reporting of the organisation

hi


Idea: Define innter table borders

The table on this page is super hard for me to read.

It feels super unclear to me which content goes in which column. Seems like the obvious solution is just to add inner table borders.

Well, I agree maybe the best option for us here is to use full borders for inner elements, but I think we got a couple of other options here. So I will briefly describe them:

We can use text-align: center; for the whole table, which makes it a little more clear but not that much.
Here how it will look:

To make a little more readable, we can add the width: 100%; to table itself, then remove the display: block; to let the table fill the available space in the row. But we should keep in mind due to display: table; nature the table won't be scrollable in the responsive view, so we have to add a media query and make its display, block once again for small devices.
Here how it will look:

To avoid many unnecessary lines and borders in the table we can just add some for dividing purposes. Well, it won't look very good at all I guess, but there are two versions of bordering here:
i. With the border gaps

ii. Without the border gaps

And the final option is bordering the whole inner elements which again got two versions:
i. With border gaps

ii. Without border gaps

And as I said earlier I think the very last one is a better and reasonable option for us here.

I think the easy answer is that we should left align those titles to match the content and make sure it feels like there's enough padding between items
Perhaps it could make sense for larger tables to have an alternating background color too

I think the easy answer is that we should left align those titles to match the content and make sure it feels like there's enough padding between items. This is likely a 'this table looks bad, but this other table looks good in this style" case.
Perhaps it could make sense for larger tables to have an alternating background color too

Yeah, this is a really good option too, I was dumb enough to not think about this one. 😁


WFLY-17552 MDC from 'default task' Thread copied to 'Batch Thread' on…

… creation but not cleared
https://issues.redhat.com/browse/WFLY-17552

@chengfang Do you know where this is copied? I don't see it being copied. I do think this is a bug, but we need to find a better place for it. We need to clear NDC too.

Yeah, clearing it inside the finally block is not enough even for batch subsystem.  Not sure if handling it centrally will work for all subsystems.  ejb subsystem already has interceptors for managing MDC NDC in ejb threads. This PR is related to a customer issue so I'll address it as a batch issue for now.

MDC data are in ThreadLocal, and are copied from the parent thread (i.e., undertow thread) to child thread when new threads are created in batch thread pool.

It really feels like there might be multiple places we need to handle it TBH. The request-controller in core, however that is not guaranteed to be set. I need to look at how all the thread pools are created and see what we can do from there. That is the ideal place is within the thread pool if that is possible.

I think the best time for such propagation is at invocation handoff time,  when you have access to both the incoming invocation data and newly created task.
Thread creation time is too early since the parent thread may not the the same as the invocation thread (a thread pool may be pre-filled). Request-controller could be too late and it does not have access to the original invocation thread.

Thanks @chengfang  and @jamezp


SAML metadata key RemoveMillisecondsFromDateTime

For SAML applications, to Remove milliseconds from the date and time, we need to use the metadata key "RemoveMillisecondsFromDateTime". As per our documentation: https://docs.microsoft.com/en-us/azure/active-directory-b2c/saml-service-provider-options?pivots=b2c-custom-policy#remove-milliseconds-from-the-date-and-time , the RemoveMillisecondsFromDateTime metadata key should be within the relying party, however, the sample snippet shows it under the 'ClaimsProvider' element which is incorrect. The metadata key for "RemoveMillisecondsFromDateTime" should be within the 'RelyingParty' element instead of the ClaimsProvider element.
Since the sample shows it under ClaimsProvider, it is confusing for the customer to test it
Incorrect method:
<ClaimsProvider>
  <DisplayName>Token Issuer</DisplayName>
  <TechnicalProfiles>
    <TechnicalProfile Id="Saml2AssertionIssuer">
      <DisplayName>Token Issuer</DisplayName>
      <Protocol Name="SAML2"/>
      <OutputTokenFormat>SAML2</OutputTokenFormat>
      <Metadata>
        <Item Key="RemoveMillisecondsFromDateTime">true</Item>
      </Metadata>
    ...
    </TechnicalProfile>

Correct method:
<RelyingParty>
    <DefaultUserJourney ReferenceId="SignIn" />
    <TechnicalProfile Id="PolicyProfile">
      <DisplayName>PolicyProfile</DisplayName>
      <Metadata>
        <Item Key="RemoveMillisecondsFromDateTime">true</Item>
      </Metadata>

Please reach out to me internally on Teams if there are any additional questions.
Thank you.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 31ad94de-b22d-b17a-8cb7-e218d7e74095
Version Independent ID: 73575d4b-b5e8-b98b-4ce3-b24c36c00db1
Content: Configure SAML service provider options
Content Source: articles/active-directory-b2c/saml-service-provider-options.md
Service: active-directory
Sub-service: b2c
GitHub Login: @kanikak-MS
Microsoft Alias: kanikakapoor

Thank you for your feedback! We will review and update as appropriate.

@kanikak-MS Thank you for pointing this out.
Yes, the RemoveMillisecondsFromDateTime metadata item key takes effect when added in the RP file. The only change I had to make in the snippet that you have shared is to move the <Metadata> element immediately after <Protocol Name="SAML2" />. Adding it after <DisplayName>PolicyProfile</DisplayName> and before <Protocol Name="SAML2" /> results in an error during upload of the RP file.
I have updated the document as per the below snippet and sent the change to the author to merge it.
  <RelyingParty>
    <DefaultUserJourney ReferenceId="SignUpOrSignIn" />
    <TechnicalProfile Id="PolicyProfile">
      <DisplayName>PolicyProfile</DisplayName>
      <Protocol Name="SAML2" />
      <Metadata>
        <Item Key="RemoveMillisecondsFromDateTime">true</Item>
      </Metadata>
      <OutputClaims>
             ...
      </OutputClaims>
      <SubjectNamingInfo ClaimType="objectId" ExcludeAsClaim="true" />
    </TechnicalProfile>
  </RelyingParty>


huge amount of packages deprecated

I'm tryin to use this, but it turns out that all packages are deprecated and throw alot of errors and warnings when installing dependencies.
Are there any plans to update this project. I find it quite unusual since the project is practically new, a couple of weeks.

I'm tryin to use this, but it turns out that all packages are deprecated and throw alot of errors and warnings when installing dependencies.
Are there any plans to update this project. I find it quite unusual since the project is practically new, a couple of weeks.

Ahoy there sir - yes there are a number of issues with the DuckDB nodejs client, including the deprecated dependencies. There is a new client that is available, and we'll be updating this to use the new client shortly.

This has been updated to use the latest DuckDB node API @rickersilva - the memory leak issues are resolved.


Add user documentation for Java Actions

Close issue #375
Add the detail guidances for Java Actions in docs/actions.md.

@psuter Please review. Thanks.

@csantanapr please review -- in particular how do we keep this doc in sync with blue docs these days?

It looks good to me as an initial description of how to get started with Java.

@csantanapr please review -- in particular how do we keep this doc in sync with bluemix docs these days?

@sjfink Once it get's review and accepted/merge in this repo, I will semi-manually sync and send a PR to  IBM-Bluemix/docs with the new updates.
There is not easy programmatically one way sync from open to bluemix docs two different formats

Updated.
@psuter @rabbah Please review.

@csantanapr @psuter Are we ready to merge it?

@psuter LGTM, review and merge when you have a chance.

Thanks everyone for the careful proofreading. @csantanapr it's in master now, if you need to work your synchronization magic.

Thanks @daisy-ycguo for the contribution.

@psuter to track sync of docs https://github.com/IBM-Bluemix/docs/issues/275
@daisy-ycguo thanks for the help


Advance to stage 3

Criteria:

[x] Those from stage 2

This is #10

[x] Complete spec text

Pending design problems being resolved.

[ ] Designated reviewers have signed off on the current spec text.

Not sure who these will be?

[ ] The ECMAScript editor has signed off on the current spec text.

Tracked by #11
Implementation types expected during this stage:

[ ] Spec compliant

Tracked as #7, #8, #9, #27, #28. Ideally this should be done at the same time as the experimental ones (i.e. there should be no experimental ones, just ones that follow the finalized spec text).
Finally:

[ ] Get TC39 to agree that we have advanced to stage 3, after meeting all the above requirements.

Do non-browsers count as implementations? If so, https://www.npmjs.com/es7-shim includes https://www.npmjs.com/package/array-includes which is one, and I believe babel may have one as well.

No


Negative correlation between number of variants and mapped reads

Dear Erik,
I have whole genome sequencing pooled data from haploid organism that I used  Freebayes to detect variants using pooled continuous and diploid options.
The steps used were:
Illumina raw data (2x150bp) --> Trimmomatic --> BWA-mem --> Picard MarkDuplicates --> GATK v.3.8.1 IndelRealigner --> FreeBayes v.1.2.0 with pooled_continuous and ploidy=2 options.
To QC the results, I plotted the number of reads vs. number of variants and I obtained an unexpected negative correlation (plots attached). The number of variants appears to reach a plateau around 20K. The two dots with low variants are control strains. The pools with lower mapped reads (low mean depth) are the ones that FreeBayes detects more variants.
Do you know what could be causing this unexpected behavior using FreeBayes?
I appreciate your help!
Barbara.

Could it be that it's easier to map reads to regions of your reference that
are common among the samples that you are analyzing? The intuition is that
lower coverage would correlate with sequence diversity that itself
frustrates the alignment.
On Fri, Aug 23, 2019, 00:17 Barbara Muller<EMAIL_ADDRESS>wrote:

Dear Erik,
I have whole genome sequencing pooled data from haploid organism that I
used Freebayes to detect variants using pooled continuous and diploid
options.
The steps used were:
Illumina raw data (2x150bp) --> Trimmomatic --> BWA-mem --> Picard
MarkDuplicates --> GATK v.3.8.1 IndelRealigner --> FreeBayes v.1.2.0 with
pooled_continuous and ploidy=2 options.
To QC the results, I plotted the number of reads vs. number of variants
and I obtained an unexpected negative correlation (plots attached). The
number of variants appears to reach a plateau around 20K. The two dots with
low variants are control strains. The pools with lower mapped reads (low
mean depth) are the ones that FreeBayes detects more variants.
Do you know what could be causing this unexpected behavior using FreeBayes?
I appreciate your help!
Barbara.
[image: Ploidy2]
https://user-images.githubusercontent.com/43009904/63553631-0c1b3d00-c509-11e9-8272-086a1a65806f.png
[image: Pooled_continuous]
https://user-images.githubusercontent.com/43009904/63553632-0c1b3d00-c509-11e9-8611-40dce69f3b58.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ekg/freebayes/issues/564?email_source=notifications&email_token=AABDQEOOXSHUAKPOPDZP5ZDQF4F7RA5CNFSM4IOZYIH2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HG46MUA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDQENBBFGRBKD5VBMYJLTQF4F7RANCNFSM4IOZYIHQ
.

Dear Erik,
I do not believe that sequence diversity justifies these results. The samples are very close to each other and the reference (<1% divergence). I still suspect something in FreeBayes might be creating this bias, where the pool with lower depth had more polymorphic SNPs (using the pooled continuous or ploidy as 2). It looks like FreeBayes is not able to distinguish sequencing errors from a true rare variant.
I ran LoFreq and CRISP for this experiment and those variant callers designed for pooled sequencing data gave me a positive correlation between number of variants and mapped reads.

Also, I extracted the variants that were common between FreeBayes and CRISP, and plotted using the FreeBayes calls. I did not observe a positive correlation, suggesting the presence of bias.

I would be interested to know if you have any suggestion to improve this behavior on Freebayes.
Thank you!
Barbara.

(I just note that you used freebayes 1.2.0 and current is 1.3.1 but not sure if that affects this particular issue)

Dear Erik,
Thank you for your answer.
My experiment is composed of pooled sequences from different unicellular cells of Chlamydomonas reinhardtii, which is haploid (n=1) organism. This project uses WGS of mutagenized algae pool population from selection experiments. Then, each library is a pool-seq from different unicellular cells that are expected to be continuous data.
I ran CRISP as a pooled discrete with poolsize=5, since my dataset has low depth. Now, I am planning to run FreeBayes using CRISP positions as target regions.
Do you recommend me to use --pooled-discrete together with --pooled-continuous?
I did not find this recommendation in your tutorial, that FreeBayes needs both options to be a true pooled called. I am confused about that. In my interpretation, it needs to be one or another option.
I was planning to rerun FreeBayes with the following parameters. Can you check and recommend the best way to run it?
freebayes
--fasta-reference         ${FASTAREF}
--bam-list                ${LIST_OF_BAMS}
--targets                 ${TARGETS}
--variant-input			${VCF}
--only-use-input-alleles
--use-best-n-alleles 4
*--pooled-continuous \ # or --pooled-discrete *
--max-complex-gap         1
--theta                   0.01
--min-alternate-fraction  0.2
--min-alternate-count     2
--min-coverage            5
--min-mapping-quality     1
--min-base-quality        10
--report-genotype-likelihood-max
> ${VCFPATH}/${FILE_CODE}_snps_and_indels_pt${SLURM_ARRAY_TASK_ID}.vcf
Dear Torsten Seemann, I don't believe it is a version issue. I will try to run in the freebayes/1.3.1 version at this time. Thanks!
I appreciate your attention.
Best regards,
Barbara.

I would suggest using both --pooled-discreet and --pooled-continuous.
They are probably misnamed, but it's hard to fix that now that there are
lots of documents referencing them.
--pooled-discrete enables pooled calling with the number of copies in the
pool defined by --ploidy.
--pooled-continuous only changes the alleles that are reported. All alleles
that pass the input thresholds are reported. Genotypes that are called
would reflect the genotyping model based on --ploidy, --cnv-map, and
--pooled-discrete
On Mon, Oct 21, 2019 at 10:43 PM Barbara Muller<EMAIL_ADDRESS>wrote:

Dear Erik,
Thank you for your answer.
My experiment is composed of pooled sequences from different unicellular
cells of Chlamydomonas reinhardtii, which is haploid (n=1) organism.
This project uses WGS of mutagenized algae pool population from selection
experiments. Then, each library is a pool-seq from different unicellular
cells that are expected to be continuous data.
I ran CRISP as a pooled discrete with poolsize=5, since my dataset has low
depth. Now, I am planning to run FreeBayes using CRISP positions as target
regions.
Do you recommend me to use --pooled-discrete together with
--pooled-continuous?
I did not find this recommendation in your tutorial, that FreeBayes needs
both options to be a true pooled called. I am confused about that. In my
interpretation, it needs to be one or another option.
I was planning to rerun FreeBayes with the following parameters. Can you
check and recommend the best way to run it?
freebayes
--fasta-reference ${FASTAREF}
--bam-list ${LIST_OF_BAMS}
--targets ${TARGETS}
--variant-input ${VCF}
--only-use-input-alleles
--use-best-n-alleles 4
**--pooled-continuous \ # or --pooled-discrete **
--max-complex-gap 1
--theta 0.01
--min-alternate-fraction 0.2
--min-alternate-count 2
--min-coverage 5
--min-mapping-quality 1
--min-base-quality 10
--report-genotype-likelihood-max

${VCFPATH}/${FILE_CODE}_snps_and_indels_pt${SLURM_ARRAY_TASK_ID}.vcf

Dear Torsten Seemann, I don't believe it is a version issue. I will try to
run in the freebayes/1.3.1 version at this time. Thanks!
I appreciate your attention.
Best regards,
Barbara.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ekg/freebayes/issues/564?email_source=notifications&email_token=AABDQEL42QFHHHNTP2BGCPTQPYH7TA5CNFSM4IOZYIH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB3XJJQ#issuecomment-544699558,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABDQEOHSPSLW6GMVDBL2XLQPYH7TANCNFSM4IOZYIHQ
.

@mullerbsf did you forget to increase --ploidy 100 as @ekg suggested?


[Cherry-pick] config: check undefined config item (#1362)

What problem does this PR solve?
cherry pick #1362

/run-all-tests tidb=release-2.1 tikv=release-2.1 tidb-test=release-2.1


Update cody commit

Let's include recent improvements from cody repo: https://github.com/sourcegraph/cody/pull/4525.
The other related PR: https://github.com/sourcegraph/cody/pull/4673
Test plan

Full QA, focused on Inline Edits

whoops, the cody commit pointed to Tom's branch instead of main - I will update it again once the his PR is merged

update cody commit to main: https://github.com/sourcegraph/jetbrains/pull/1812


Local build doesn't work

Hello,
If i try to build the solution locally, i get Sailaway API errors. But the downloaded version works properly. Is the source on GitHub up to date?
Thank you.

That's because you need an API key.
You need to input the API key here: https://github.com/expilu/sailaway-api-to-nmea/blob/91a86e9f55d0b7e3524efc22f6b3778c13ebc743/SailawayToNMEA/App/Conf.cs#L6
The version I distribute has an API key provided to me. You need to reach the developer of the game to have your own.


Word spacing update

Hi @twlamb, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, DNFBOT;

@twlamb  thanks.


Add ability to use ssh vs https vs http to download repositories

I need to vendor my own project repositories (for instance, repository 'api' which shared between frontend and backend).
For these repositories I want to specify clone url like<EMAIL_ADDRESS>instead of https:// scheme which ask my username + password every time when I do something with dependencies (or on simple checkout)

@zamotivator agreed there should be a solid plan and pattern to handle this case.  Its one of the reasons I started govend, so I could selfishly include my companies private repos: https://github.com/govend/govend/blob/9a085218c2d461217e35df2df4cebec657db72bc/deps/vcs/vcs.go#L863-L869
In your case though, would a simple netrc file give you what your looking for?

Here is some documentation for .netrc that you might find helpful in the short term:

https://confluence.atlassian.com/bitbucketserver/permanently-authenticating-with-git-repositories-776639846.html
https://gist.github.com/technoweenie/1072829

@zamotivator any thoughts on the comments above?  Has the .netrc file solved your issue?

@jackspirou I'd prefer not to use .netrc as I do not like to store my credentials on my FS unencrypted, especially to Github.

@jackspirou I'd prefer not to use .netrc as I do not like to store my credentials on my FS unencrypted, especially to Github.

You can also use a token which prevents leaking your real credentials.

@JohnMaguire as @tboerger mentioned above, you can use a github personal token, in lieu of your password within the .netrc file.
I think this is an issue worth noting, given the .netrc option, it shouldn't stop anyone at the moment.
I would like to know how go get behaves around private repositories, and private GHE servers.  Does it start with ssh and then default to https and then to http?

Thanks, I did not realize that. I feel much better about using an access token.
As far as handling private repositories go, there is some interesting discussion here: https://gist.github.com/shurcooL/6927554
I also just now realized (d'oh!) the behavior that I thought I was seeing: govend defaulting to HTTPS regardless, is actually because I had a typo in my package name. I am not yet running into the issue seen here. It does however look like the above Gist might contain a workaround.

There is a way to force git to always use ssh: https://www.quernus.co.uk/2015/10/22/force-git-to-use-ssh/


Feature: avoid write-write conflict errors for single document updates

When using RocksDB, two concurrent update operations on the same document can result in a write-write conflict (error-code 1200). This type of error often causes confusion among users (this is a frequently asked question in Slack) and puts additional burden on the developer - one has to be aware of this error and how to handle it. I know that it is possible to use exclusive mode, but this locks the whole collection, which is overkill in many cases.
However, I think it should be possible to avoid this error for almost all operations that update only a single document.
At the moment, a developer has two choices to handle this error: either propagate it, or perform a retry. IMHO propagating it is hardly ever the right choice - if we have performed an unconditional update (i.e., not specified a specific document revision), then I simply want this update to be applied - regardless of document's state and any other potential concurrent updates. Performing a retry causes additional overhead during development (has to be implemented) and runtime.
The main point is that the error is not deterministic - if the requests are processed with slightly different timings (such that they do not overlap), then the updates are simply applied without any error.
So what it comes down to: If there are multiple concurrent updates for the same document, then they should be serialized in some way.
I would like to trigger a discussion about if and how this could be improved.

impossible to do without adding the retry logic inside the database. Especially considering that we do not know the intention of the developer.

I think to free developers from the burden of retrying or conflict handling, IMHO there are at least the following options:

serializing writes to (specific) collections automatically, regardless of whether keys conflict. This would be an easy solution, both to implement on the server side and for developers to use, but would pessimize writes for which there are no conflicts.
retrying writes on conflict automatically for transactions/operations that are flagged as such. This would require minimal changes in client applications, and allows fine-grained control of when to retry (and when to fail with conflicts).

Do you see any further options?

Well, as a developer, if I send a request to update a document in the DB, I expect that update to be applied. If I want to update a specific revision, then I can set ignoreRevs to false, and if the specified revision is no longer valid, then reporting a conflict error is perfectly fine.
But when I don't specify a revision, I simply want the update to be applied unconditionally.
AFAIK the RocksDB engine acquires a write-lock on a to-be-updated document. If this fails (because another request is currently updating the document), you abort with said error. Wouldn't it be possible to simply wait for some (limited) time until we could acquire the lock, so we can perform the update?

If a patch operation only changes attributes of a document that the other operation didn't change, this could qualify for retrying such an internal update retry:
{
 a: true,
 b: false
}

Update 1 flips a), Update 2 flips b) - one could argue here that these don't conflict.

@mpoeter : for a single document update operation, we don't acquire a lock inside our storage engine wrapper, but RocksDB internally will track all modifications and locks the underlying keys.
So two concurrent operations that modify the same key may interfere on the RocksDB level.
The non-trivial thing here is that a single document operation is not a single write to RocksDB internally, but multiple writes. And any of these writes can conflict.
IMO a potential solution is to abort the update altogether, and restart it from scratch, until it either succeeds or runs into a timeout.
For single document operations this might actually work.
For multi-document transactions one would need to restart the entire transaction, with its past operations potentially already unknown, or at least not readily available to the server for re-application.

@jsteemann Thanks for the explanation! I wasn't aware that a single update requires multiple writes (but it makes sense now that I think of it, since you have to update indexes etc).
In this case, aborting the operation and restarting it from scratch sounds like a good approach. But of course this cannot be applied to multi-document transactions.

the retry is basically unnecessary. If you don't care which write wins one could just pretend everything was fine and let the other write "win". Retrying just wastes CPU cycles where multiple threads compete to be the "last" one.

No, because many "conflicting" operations can be combined without destroying the effects of the other operations. @dothebart already gave an example - two patch operations that change different attributes of the same document. But there are also queries such as:
FOR doc IN docs
  FILTER doc._key == ...
  UPDATE doc WITH { counter: doc.counter + 1 } IN docs

FOR doc IN docs
  FILTER doc._key == ...
  UPDATE doc WITH { list: PUSH(doc.list, "blubb") } IN docs

In these cases we cannot pretend that everything was fine, because our changes are missing!

This whole thing is usually called transaction coordination and I fully agree with @mpoeter that it is not the developer's job to write retry logic and pray that requests go through. Frequent retrying by developers, even if perfectly written (typically not the case if done externally), will most probably result in key congestion and clog the DB with a lot of unnecessary requests which intermittently or ultimately fail, or as @graetzer graetzer puts it, "just waste(s) CPU cycles".
Think of a social network where a successful post suddenly receives a ton of likes around the same time. These likes are pushed into the DB from distinct instances and boom, the DB clogs up and only a small percentage of the likes succeed. I am new to Arango and I can see you guys are doing a lot and many approaches are superior to other DBs (and maybe I am saying something here that Arango already is capable of), but not having internal transaction coordination renders Arango useless for social media, the mother of all graph applications.
A good transaction coordinator looks at read/write requirements of a job to schedule it either asynchronously or synchronously among the other jobs. If a job fails and a retry becomes necessary, we have learned why it failed (which keys made it fail), which results in vastly improved schedulabilty (does that word even exist?) for the second execution. If retrying is left to the developer, the only thing the developer can do is push the job again, hoping that it will succeed next time.
According to https://www.arangodb.com/docs/stable/transactions-transaction-invocation.html#execute-transaction collections must be specified when beginning a transaction. IMHO, this doesn't make so much sense anymore for Rocks and it would rather be useful to optionally specify the document keys to be written instead whole collections.
db._executeTransaction({
	action,
	collections: {
		read: ['users'],
		write: ['posts', 'has_liked'],
	},
})

I am further seeing above as optional optimization hints, which the developer can specify for improved scheduling of critical jobs.

If specified, the transaction coordinator can perfectly queue the job by key(s) and prevent conflicts with other jobs.
If not specified, the first execution would be async with other jobs. When it fails, it is inserted into a specific queue for the document key which made it fail, and processed in sync with other jobs for the same key.

To keep it simple, I would write a transaction coordinator which has

a generic queue (for the async jobs that we have right now, and which will still be executed async), plus
one retry queue for each (failed) key (or combination of keys). In regular intervals, the queues are flushed (and if there are too many collisions during async processing, the generic queue is stopped for a moment so the document queues can be emptied without further collisions).

I just hit this error running a migration - I had two separate update queries that hit the same documents.
I don't have anything valuable to add to the conversation, but I'm suddenly very concerned about my decision to go with ArangoDB. The possibility that two connected clients fire off updates to the same document at the same time seems like entirely normal behaviour. Are we saying that these can randomly fail in Arango, and it's my responsibility as the app developer to detect the error and retry? Surely that would apply to every place I do an UPDATE?

not having internal transaction coordination renders Arango useless for social media, the mother of all graph applications

If this is true, it seems like narrowing the issue to social media is understating things considerably. Any app can potentially have a flurry of activity around the same document.
Should I be worried?

Obviously I (as the original author) have a very strong opinion regarding this issue. I have opened this ticket when I was just a normal user, but in the meantime I have joined the ArangoDB team. Unfortunately I have other things to work on right now, but I still have this on my personal todo list. In fact, I have discussed this with one of my colleagues and could convince him to look into this (at least for single document operations via the document API). I am optimistic that we will fix this eventually, but it might take a little.

@mpoeter are you able to offer any advice on what this practically means for application code? Worst case scenario it sounds like I need retry logic around every place where I write to the db. But as you mention above, such logic will probably cause problems of its own.
I'm struggling to see why a proper solution wouldn't be a must-have pre 1.0 feature. But I'll assume I'm wrong about that given Arango is obviously doing well. Is there some reason why, in practice, this is not as bad as it sounds?

We are migrating to MySQL

@mpoeter :

However, I think it should be possible to avoid this error for almost all operations that update only a single document.

Does that mean, even with the fix, operations that update more than one document will not have this protection?

I have been running into this more and more frequently. I have a very active database with frequent updates from many sources and sometimes the write-write conflicts are becoming persistent due to the backlogs of writes. I do love ArangoDB and the AQL which makes it much easier to write complex queries but this problem is starting to become a major problem.
It looks like that ArangoDB is not well suited in a multi-threaded system because of this behavior. And if I wanted to make it work, I need to channel all queries that have write operations through a single process as a queue.
Why can't ArangoDB do this automatically? During the parsing of queries it could easily determine whether the query contains write operations and which collections it modifies, and use queues for each collection for these queries - executing them sequentially. Sure, it would be a bit slower and queries might time out but it's still a lot better than randomly failing!
Could we get some manpower behind finding a solution to this?
This problem has been known for at least 3 years, but not fixed!

Like @TehShrike we've abandoned Arango for this reason.

Same here, abandoned for the same reason. Arango simply does not scale. It's a toy, not a production DB.

I prefer not to abandon ArangoDB because it has many features that I love and AQL is just superior to SQL.
I did some further searching around and found that there is an option which does essentially the same as what I proposed.
For all operations that modify a collection, there is a exclusive option which locks the collection for writing for the time of the update, insert, etc operation so that document level write-write conflicts can't happen.
Eg: https://www.arangodb.com/docs/stable/aql/operations-update.html#exclusive
Now, I'm going to add this option to all of my queries with write operations and see what happens.
If that works and doesn't slow things down then I'll propose to make this the default, and if someone want to play "fast and loose" then they can turn if off.

Afaik, it locks the complete collection database-wide, not just the document you write. It has been a long time that we switched, but as I recall, this won't scale. You'll avoid the conflicts, but requests are queued after each other and the request global (!) queue becomes longer and longer. We had the request queue take minutes to resolve after no longer sending requests. 3 powerful servers were at the load limit. When we switched to CockroachDB, same load, non-optimized SQL statements, load was at less than 5%, same hardware. My recommendation: trash Arango for something that scales. It just won't work. For several years I hoped it would, but it simply doesn't. [I'm not related with CockroachDB in any way.]

Here's the result of my experiment: success. After adding options {exclusive:true} to all queries which made modifications has eliminated the write-write conflict error entirely. So far nobody has noticed any discernible reduction of speed due to this change.
So I would recommend anybody running into this issue to try adding the exclusive option to the queries. In some use cases it might slow things done too much, as in @oliverhausler 's case but it could be a workable solution in other cases.

Thanks for testing and sending results. It's what I had expected. It works until you reach a certain limit and the queue starts building up. We had simulated 5,000 simultaneous clients, but the number isn't fix as it depends on query complexity. Problem is that collections are locked globally (e.g. only the master can process writes), which means it makes no difference if you have 1 server or 1,000. As soon as you reach max throughput, your system can't scale further.
Anyhow, good luck with your setup, Laszlo.

This is a feature that me and the team I work with are interested in. There is a part of our system that is prone to receive small burst of updates on the same document. We started getting this error since the switch to RocksDB engine.
We handled it by implementing the retry pattern with a backoff value that varies for each actor. If all retries fail, we then use OPTIONS { exclusive: true }.

This is an issue my team also has several times per day (around 100 times).
We are going to implement retry on our side since it seems to be the quickest way to fix it, but we do believe the issue should be fixed on the DB side.

In arangojs v7.8.0, retryOnConflict option was added to db.query.
Compared to @DiscoPYF's comment https://github.com/arangodb/arangodb/issues/9702#issuecomment-1160790241, it lacks backoff and a final try with OPTIONS { exclusive: true }.
retryOnConflict will simply requeue the task if it failed because of a write-write conflict until the number of allowed retries is exhausted.

@DiscoPYF , your handling seems like a good one that preserves the performance while still can handle the conflict.
The only things it's a layer of complexity that needs to be added to the handling of queries. In my opinion, this handling should be the responsibility of the database engine. When I issue a query to a database engine, I would expect it to execute it and only fail if there's a critical error that cannot be handled automatically.

Any real update on this?

I wanted to provide a quick update. I am currently working on a first version that solves this for single document operations performed via the document API (i.e., not AQL). This should be available in the next version (3.11) and will most likely not be backported to older versions.
We also have ideas how to generalize this for batch operations as well as AQL and streaming transactions, but there is no decision yet if/when this will be implemented.

I keep hitting this error at random in my backend. Using arangojs, it would be great if there could be a fix for it. I'm just pushing single document update and it happens at random :/

Subscribing to the issue, since it's a major pest: my backend can do parallel writes to the same document, but different fields in them (guaranteed by logic), by consuming the changes in parallel from Kafka.
Somehow in 3.9 the issue became much more prevalent, 3.10 has some weird view consistency issue (to be researched deeper) and doesn't fix anything anyways.
The update says "maybe 3.11" - looking forward to it.

I've got update on how I'm reproducing the issue.
I run
1 docker backend nodejs/arangojs
2 docker arangodb
docker-compose, using volume mapping to map arangodb3/arangodb-apps to local disk (windows)
The write-write happens with 1 user doing 1 write 90% of the time, mostly via collectionPtr.replace() and update() too using arangoJS.
Hope this maybe help ?

The PR mentioned above improves the situation for single document operations performed via the REST API. It currently does not support AQL, streaming transactions, or batch document operations. We now serialize operations on the same document by locking the documents key. To avoid starvation this lock acquisition time is limited to 1s. If the lock cannot be acquired within this time, a write-write conflict will be returned as before, but it is now much more unlikely. The lock timeout is fixed for now, but will be made configurable in the future.

We were excited to use ArangoDB and began dual writes between our old system (Google Firestore) and ArangoDB (ArangoGraph) when we hit this issue immediately. We'll come back when 3.11 is available there and see how things go then. I'm really glad you're working on it @mpoeter. It is so strange to me that ArangoDB can be a production database with this issue.

Does Speedb improve this limitation of Rocksdb? https://github.com/arangodb/arangodb/issues/18039

Hello!
@mpoeter It's really nice that updating a single document has bean fixed, but what about multiple document updates? I'm had this issue in my migration that adds new fields in the collection?
Unfortunately OPTIONS { exclusive: true } don't help me. With this option, migration takes too long to finish (more than 1 hour with about 1.000.000 documents).

We have an AQL to run in C# similar to the following pseudo code called within a transaction. Each transaction is wrapped within a unit of work, and multiple instances of the same unit of work are spawned in parallel by C# tasks. Eventually some of these AQLs receive write-write failure and their transactions fail. Implementing a backoff policy in place to retry after a few times with 2,4,8 and 16 seconds delay did not help recover from the write-write race. Even using exclusive: true was not conclusive too.
FOR c IN [{"_key":"2","Version":0}]
INSERT { _key: c._key,  Version: c.Version+1 } INTO MyCollection OPTIONS { overwriteMode: "update" }
RETURN { Id: c._key,, Version: NEW.Version }

We don't know what the next step should be to address this situation properly to make the code work in a multi-thread scenario.

Any update ?

In the documentation here, under locks section, I see
"However, if two concurrent transactions attempt to modify the same document or index entry, there will be a write-write conflict"
Consider, I have a collection named food with below two documents and an index on the fields "category, color".
food:
{
name: apple,
category:  fruit
color: red
},
{
name: raspberry,
category: fruit
color: red
}
if I have a transaction T1 modifying the first document(apple) and another transaction T2 modifying the second document(raspberry) concurrently. Though they are different documents, can i get write-write conflict because they have same value of for category and color on which the index is built?


Test & parenthesis

grounding:

formula:
psi := \top.

program:
test := ?a \and b?.

main:
psi.

output : "Fatal error: exception Invalid_argument("Parser error. In file issues/issue3.pa, at line 7, column 11: syntax error. \n (error 151)")"
expected : no errors
Using "test := ?(a \and b)?" will solve the problem

Solved in https://github.com/AbdallahS/dlpag/commit/e0b4980ed6208c86b97414fa82ca75a8990821f7


Scrolling down needs faster input than scrolling up

Environment data

VS Code version: 1.58.2
Jupyter Extension version (available under the Extensions sidebar): v2021.8.1054968649
Python Extension version (available under the Extensions sidebar): v2021.7.1060902895
OS (Windows | Mac | Linux distro) and version: Windows 10 Version: 10.0.19043 Build 19043
Python and/or Anaconda version: Python 3.8.9
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Jupyter server running: Local

Expected behaviour
Scrolling in jupyter notebooks should be easy and responsive.
Actual behaviour
Scrolling up is fine, but when trying to scroll down, its as if there is a threshold for speed. If I scroll with slow hand movement, it doesn't pick up the movement of the hand(two finger scroll) However when scrolling up, it works fine.
Steps to reproduce:
Just create enough cells and blocks in a jupyter notebook and try scrolling up and down slowly on a trackpad, you will see the difference.

Hi @nivibilla, are you able to provide the notebook you are experiencing scrolling issues with?
Could you follow the steps below to get us info that will allow us to improve the scrolling:

Have notebook you are experiencing issues with open

Open command palette (Ctrl + Shift + P or Cmd + Shift + P) and select "Developer: Toggle Developer Tools"

On the top click "Performance". If you don't see "Performance" it may require you click the ">>" button

Click record and scroll up and down within your notebook and hit stop when you feel you have captured the lag.

On the top click "Save Profile" and please upload to the issue.

If you are able to share, uploading the notebook you are having issues with would be especially helpful.


[Removal]:

Hotspot Name
Winning Syrup Canary
Hotspot b58 Address
Winning Syrup Canary
Discord Handle
No response
Hotspot Manufacturer
Heltec Automation
Removal Reason
Hello, it's a real deploy. Could you please remove it from denylist? Thank you in advance. Regards
Modifications
No
Extra forwarders
No
Extra antennas
No
Additional Information
No response

We have taken into account this removal request in the next iteration of the denylist. It may take some time before it's processed. This is an automated comment due to the volume of addition and removal requests. A future tool is in the works to provide more insight into the analysis approach made by the Helium team and community members that maintain this list.


default chat template method fix

This PR addresses issue https://github.com/EleutherAI/lm-evaluation-harness/issues/2231, which causes a TypeError: 'str' object is not callable when using non-HF models. This error was introduced following the merge of PR https://github.com/EleutherAI/lm-evaluation-harness/pull/2129

Hi! Thanks for this! won't quite fix the error for the other models as they still use chat_template's previous signature. We could move the chat_template method from HF to the LM base class and have all models inherit. Would need a condition like:
if not hasattr(self, "tokenizer") or self.tokenizer is None:
    return None

at the beginning to handle the case where the model might not have a tokenizer. Thoughts @haileyschoelkopf?

Thanks for pointing that out! I’m happy to make the modifications or update the signatures in other models as needed. I’ll wait for @haileyschoelkopf’s decision

Let's change the signature+docstring in the LM class to make it consistent across models (but leave in LM as returning an empty string), and add the specific HF-tokenizers-like chat_template implementation in TemplateLM, which VLLM and HF can inherit from. How's that sound?
Caveat to this: this makes TemplateLM expect to have a self.tokenizer attribute (which we did not previously require) so there should be an escape hatch:
if not hasattr(self, "tokenizer") or self.tokenizer is None:
    return None

at the start as @baberabb mentions! (possibly w/ error log when tripped: "lm.chat_template() method inherited from TemplateLM but LM has no self.tokenizer attribute, please reimplement lm.chat_template to log the template used if the template used is configurable!" or something of this sort)
Prefer not to add a mixin at least for now, I think, though willing to be convinced otherwise!

@haileyschoelkopf Thanks for the update. The Mixin suggestion from the other issue is to have all chat-related code in one place soon.  One can already see this code starting to be everywhere, and I think this makes any major change prone to errors.
e.g.,
https://github.com/baberabb/lm-evaluation-harness/blob/ca3d86d6b8dea86211ec60b93c3c026ce73c9d60/lm_eval/models/api_models.py#L44
https://github.com/baberabb/lm-evaluation-harness/blob/ca3d86d6b8dea86211ec60b93c3c026ce73c9d60/lm_eval/models/api_models.py#L169
https://github.com/baberabb/lm-evaluation-harness/blob/42dc244867889a19ae80847254a481f446f6e4b7/lm_eval/api/samplers.py#L66
But since this is blocking an actual bug, I don't want to delay this PR.

I think the mixin suggestion makes sense and would be in favor of it! I just want to make sure that in the process we don't cause an LM subclass someone has written externally to end up failing due to the complete lack of these methods if they don't inherit themselves from the new mixin.

continued in #2269 , thank you @KonradSzafer for starting on this!


resources quota setting for container creation

Hello!
Added NanoCpu and Memory parameters to HostConfig – see ContainerCreate docs.

Hi @AlexPtnn, thanks for raising the PR! Were you expecting so many reactions on it? 😄
Looks like the build is failing because this feature isn't supported when running rootless. This is fine, you can add a condition in the test as done here. Maybe put a note in the docs pointing it out.
FYI I will be looking to add the concept of modifiers to tc-node, as it exists in other langs, so that we don't need to keep adding code for such cases. So it's possible this will be deprecated quite quickly, but will be available via the modifiers.

@cristianrgreco fixed tests. Added note to FAQ.
Can you please explain what you mean by "concept of modifiers"? I'm not familiar with this concept 🤔
This feature is blocker for us, because we have limited resources for containers in our test environment by default. So I need to explicitly pass quota for my containers to start in such conditions.

@AlexPtnn Yep I should be able to help out with a fast release. There's one more change to make, the method needs to be added to the TestContainer interface: https://github.com/testcontainers/testcontainers-node/blob/4ce80ac7797b320f34d4f18048f3f99c49eb124d/src/test-container.ts#L18

the method withResourcesQuota needs to be added to the TestContainer interface

made it

Be sure to export any types necessary (ResourcesQuota?) in index.ts.

I don't see any types (src/docker/types.ts) exports from index.ts. I exported my type inside types.ts like other types exported in this file. That's enough I think. Did I missed something?

expose modifiers that let users "go down a level" and configure the underlying request to Docker

I think you can just expose dockerode HostConfig interface by method like this:
withHostConfig(config)

@cristianrgreco Hi! Thanks for merging my MR.
Can you please publish a new release as we discussed earlier?
Do you have automatic release cycle in other langs? Can we make one for tc-node? Any limitations for it?

@AlexPtnn Released in 9.3.0.

Do you have automatic release cycle in other langs? Can we make one for tc-node? Any limitations for it?

There is already a pipeline for publishing, it is just a case of publishing a release.

@cristianrgreco thanks! Works great in out project!
"automatic release" – I mean like auto release after merging MR. Minor/Major version update according to commit message. Or you have some limitations on pipeline resources, extra payments? I'm not familiar with Github tariffs.


Fix some caching typos and consolidate kernel startup code

Fixes #8580
This PR fixes some small mistakes in caching and consolidates kernel startup code into as single execution.
However Conda just takes a long time to start.
For me on an environment with Keras in it (the slowdown of activation)

10 second to search registry to find where conda is
10 seconds to create the keras enabled environment and pull out its environment variables
4 seconds to actually launch the kernel

If caching is working, the first two steps happen in the background and shouldn't be noticeable for most users.

Codecov Report

Merging #8674 (71bbf28) into main (8b30916) will decrease coverage by 0%.
The diff coverage is 91%.

:exclamation: Current head 71bbf28 differs from pull request most recent head 3e50c28. Consider uploading reports for the commit 3e50c28 to get more accurate results

@@          Coverage Diff          @@
##            main   #8674   +/-   ##
=====================================
- Coverage     71%     71%   -1%
=====================================
  Files        384     384
  Lines      24634   24651   +17
  Branches    3791    3792    +1
=====================================
- Hits       17635   17615   -20
- Misses      5468    5497   +29
- Partials    1531    1539    +8

Impacted Files
Coverage Δ

src/client/telemetry/index.ts
63% <ø> (ø)

src/client/datascience/jupyter/kernels/kernel.ts
75% <85%> (-2%)
:arrow_down:

src/client/common/application/workspace.ts
82% <100%> (ø)

src/client/common/process/condaService.ts
79% <100%> (+1%)
:arrow_up:

...ent/common/process/environmentActivationService.ts
78% <100%> (+<1%)
:arrow_up:

src/client/common/process/proc.ts
72% <100%> (ø)

...ient/datascience/editor-integration/codewatcher.ts
70% <100%> (+<1%)
:arrow_up:

...t/datascience/notebook/vscodeNotebookController.ts
80% <100%> (+<1%)
:arrow_up:

src/client/datascience/dataScienceSurveyBanner.ts
69% <0%> (-7%)
:arrow_down:

...datascience/editor-integration/codelensprovider.ts
78% <0%> (-6%)
:arrow_down:

... and 21 more


Gallery not properly styled on the website

Issue Summary
When adding a gallery to post, it doesn't get styled properly on the website. Every photo are at their max height and constrained by post content width.
To Reproduce

Add a gallery
Go on preview or published post
See unstyled gallery

Can be seen here: https://blog.tribolet.fr/p/caac7644-d323-4ae4-93c2-19bacaf3dcd0/
https://screenshots.firefox.com/V85Omqxk14VAhBKA/blog.tribolet.fr
Technical details:

Ghost Version: 2.1.1
Node Version: v6.14.4
Browser/OS: Firefox 63.0b4 (64-bit) and Chrome 69.0.3497.81 (Official Build) (64-bit) / Windows 10
Database: mysql

@ltribolet based on that post it looks like you're using Casper, which should have image gallery support. However, I don't see any of the styles for the gallery images in your theme's CSS, which leads me to believe you're using an outdated version of Casper or a fork of it. Either way, you will need to update to the latest version of Casper in order to get the changes necessary for galleries.
Going to close this as the issue should be fixed with an update of Casper (Use the forum for help if you run into issues updating Casper). If you find that the issue is not fixed after updating Casper, we can reopen this and investigate further 😄

Do I need to manually update Casper theme even while using ghost update? Because I don't remember doing any customization to this theme.


Fix for Bug #449

This has already been fixed in the development branch via https://github.com/absolute-quantum/cats-blender-plugin/pull/382
As a side note, I recommend you squash your commits when they're all part of the same change.


Different decorating

Is there any opportunity to apply different styles on days (i.e. Monday - white, Tuesday - red, etc.)? I was digging into using decorators but it seems they provide an opportunity to apply only one style to selected days.

You had the right idea, you can check if the day is a Monday - Tuesday - etc, in your decorator and apply the right style to it.
// Decorator for Mondays. Do the same thing for each days of the week.
public class MondayDecorator implements DayViewDecorator {
    private final int color;

    public EventDecorator(int color) {
        this.color = color;
    }

    @Override
    public boolean shouldDecorate(CalendarDay day) {
        // Check if it's a Monday here and return true if it is. I let you choose the way you want to check this.
        // For example:
        return YourDateUtils.isMonday(day);
    }

    @Override
    public void decorate(DayViewFacade view) {
        view.addSpan(new DotSpan(5, color));
    }
}