bug: Genotype structuration error

Description of the bug

Genotype structuration by variants is failing
To Reproduce

I followed the steps described in the tutorial using my own data and building the same file architecture i.e.
.
├── annotations
├── genotypes
│   ├── samples
│   └── variants
└── variants

Everything worked perfectly until the following :
mkdir -p genotypes/variants/
input=$(ls genotypes/samples/ | xargs -I {} -x echo "-i genotypes/samples/"{} | tr '\n' ' ')
variantplaner -t 8 struct $(echo $input) genotypes -p genotypes/variants/

Full traceback

Full traceback
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/usr/local/lib/python3.9/site-packages/variantplaner/struct/genotypes.py", line 40, in __hive_worker
    lf = normalization.add_id_part(polars.concat(lf for lf in lfs if lf is not None))
  File "/usr/local/lib/python3.9/site-packages/variantplaner/normalization.py", line 84, in add_id_part
    return lf.with_columns(id_part=polars.col("id").variant_id.partition())  # type: ignore # noqa: PGH003
AttributeError: 'VariantId' object has no attribute 'partition'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/variantplaner", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/variantplaner/cli.py", line 390, in struct_genotypes
    struct.genotypes.hive(input_paths, prefix_path, threads=threads, file_per_thread=file_per_thread)
  File "/usr/local/lib/python3.9/site-packages/variantplaner/struct/genotypes.py", line 113, in hive
    pool.starmap(
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
AttributeError: 'VariantId' object has no attribute 'partition'

Expected behavior

Spliting of the genotypes information in a hive like structure in the genotpes/variants folder
Environment information

variantplaner --debug-info  # | xclip -selection clipboard

System: Linux-4.18.0-513.9.1.el8_9.x86_64-x86_64-with-glibc2.28
Python: cpython 3.9.18
Environment variables:
Installed packages:

variantplaner v0.2.4

Hi,
Badly I can't reproduce your bug, are you sure you have 0.2.4 version of variantplaner.
Can you perform a full reinstall? (sorry for install time).

Hi,
Thanks for your answer.
It could be that my rust-toolset was broken/misinstalled...
I just started the re-installation of variantplaner 0.2.4 and will let you know how this goes.

Just remove the last /
run:
mkdir -p genotypes/variants/
input=$(ls genotypes/samples/ | xargs -I {} -x echo "-i genotypes/samples/"{} | tr '\n' ' ')
variantplaner -t 8 struct $(echo $input) genotypes -p genotypes/variants

Hi @natir,
Sorry for the delay in reply.
I tried removing the trailing / but it stills ends up with the same issue...
So I ran the command in verbose mode as the following :
variantplaner -t 8 -v -v -v -v struct $(echo $input) genotypes -p genotypes/variants 2> vp.errorlog

I also send you the vp.errorlog produced by the above command.
Looking at it, I wonder if this could come from my sample names : they all have _ in their names.
As the sample names are used to build the path of the .parquet files, this could cause issues in src/varinatplaner/struct/genotypes.py when joining all paths by _ :
basenames = ["_".join(p.stem for p in g_paths if p is not None) for g_paths in path_groups]

--> variantplaner could be confused when trying to get each path back from path_groups, right ?
(I cand send you a smaller subset of data for more convenience if you wish)
On my end, I will try resetting my data lake with sample names without _ in their name.

Nope, still the same error, even when having sample names without _...
I keep searching !

I think I spotted something.
Let me first explain what I did :
I cloned your mained branch, added some error message in src/variantplaner/struct/genotypes.py, line 46 :
43    for name, df in lf.collect().group_by(polars.col("id_part")):
44         # required for python 3.11 and polars 0.20.4 wired
45         fix_name = name[0] if isinstance(name, tuple) else name
46         logging.error("trying to write : " + os.path.join(output_prefix, f"id_part={fix_name}", f"{basename}.parquet"))
47         df.write_parquet(output_prefix / f"id_part={fix_name}" / f"{basename}.parquet")

Then I reinstalled variantplaner and ran the command with only one sample (to minimize the size of the output) :
variantplaner -v -v -v -v -t 8 struct -i genotypes/samples/2081345853.parquet genotypes -p genotypes/variants

Traceback
2024-01-17 12:43:00,026 - main:DEBUG: parameter: threads=8 verbose=4 debug_info=False
2024-01-17 12:43:00,028 - struct:DEBUG: parameter: input_paths=(PosixPath('genotypes/samples/2081345853.parquet'),)
2024-01-17 12:43:00,030 - struct.genotypes:DEBUG: parameter: prefix_path=PosixPath('genotypes/variants')
2024-01-17 12:43:00,030 - struct.genotypes:INFO: paths=(PosixPath('genotypes/samples/2081345853.parquet'),) output_prefix=PosixPath('genotypes/variants'), threads=4, file_per_thread=15
2024-01-17 12:43:00,205 - struct.genotypes:INFO: path_groups=[(PosixPath('genotypes/samples/2081345853.parquet'), None, None, None, None, None, None, None, None, None, None, None, None, None, None)], basenames=['2081345853']
ERROR:root:trying to write : genotypes/variants/id_part=15/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=111/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=22/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=34/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=0/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=49/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=31/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=140/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=155/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=117/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=36/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=156/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=159/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=63/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=12/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=141/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=43/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=84/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=25/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=78/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=138/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=143/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=7/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=108/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=99/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=46/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=42/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=87/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=135/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=75/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=144/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=57/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=18/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=114/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=152/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=54/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=123/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=48/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=9/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=93/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=27/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=1/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=4/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=45/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=40/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=37/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=153/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=81/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=58/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=3/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=21/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=126/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=33/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=137/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=149/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=134/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=19/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=39/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=129/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=164/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=120/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=51/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=52/2081345853.parquet
ERROR:root:trying to write : genotypes/variants/id_part=None/2081345853.parquet
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File<EMAIL_ADDRESS>line 47, in __hive_worker
    df.write_parquet(output_prefix / f"id_part={fix_name}" / f"{basename}.parquet")
  File<EMAIL_ADDRESS>line 3496, in write_parquet
    self._df.write_parquet(
FileNotFoundError: No such file or directory (os error 2)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/variantplaner", line 8, in <module>
    sys.exit(main())
  File<EMAIL_ADDRESS>line 1157, in __call__
    return self.main(*args, **kwargs)
  File<EMAIL_ADDRESS>line 1078, in main
    rv = self.invoke(ctx)
  File<EMAIL_ADDRESS>line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File<EMAIL_ADDRESS>line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File<EMAIL_ADDRESS>line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File<EMAIL_ADDRESS>line 783, in invoke
    return __callback(*args, **kwargs)
  File<EMAIL_ADDRESS>line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File<EMAIL_ADDRESS>line 390, in struct_genotypes
    struct.genotypes.hive(input_paths, prefix_path, threads=threads, file_per_thread=file_per_thread)
  File<EMAIL_ADDRESS>line 117, in hive
    pool.starmap(
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib64/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: No such file or directory (os error 2)

You can see that it tries to write in genotypes/variants/id_part=None/2081345853.parquet at one point.
But the folder genotypes/variants/id_part=None does not exists.
Is it supposed to create this id_part=None folder ?

If variantplaner try to create an id_part=None flooder it's mean you have some variants with variants_id equal to None.
Can you verify that this is the case (via pqrs or duckdb) if there are indeed variants_id with Null values. The chromosome is probably not present in the chromosome size csv file.

Indeed !
>>> duckdb as db
>>> db.sql("SELECT * FROM read_parquet('/mnt/datagenetique/data/BIT/PRODUCTION/Databases/BMW/test/variants.parquet') WHERE id is NULL ORD
ER BY chr")
┌────────┬─────────┬───────────┬─────────┬─────────┐
│   id   │   chr   │    pos    │   ref   │   alt   │
│ uint64 │ varchar │  uint64   │ varchar │ varchar │
├────────┼─────────┼───────────┼─────────┼─────────┤
│   NULL │ 1       │  62577325 │ G       │ A       │
│   NULL │ 1       │ 160351691 │ C       │ T       │
│   NULL │ 1       │  26469248 │ G       │ A       │
│   NULL │ 1       │ 197146145 │ A       │ C       │
│   NULL │ 1       │   1756352 │ C       │ T       │
│   NULL │ 1       │ 161305670 │ T       │ G       │
│   NULL │ 1       │  68431531 │ G       │ T       │
│   NULL │ 1       │  43621160 │ C       │ T       │
│   NULL │ 1       │ 204245635 │ G       │ A       │
│   NULL │ 1       │ 207023129 │ C       │ T       │
│     ·  │ ·       │     ·     │ ·       │ ·       │
│     ·  │ ·       │     ·     │ ·       │ ·       │
│     ·  │ ·       │     ·     │ ·       │ ·       │
│   NULL │ 1       │ 173659397 │ C       │ G       │
│   NULL │ 1       │ 169710478 │ A       │ G       │
│   NULL │ 1       │  31756678 │ C       │ T       │
│   NULL │ 1       │  13308568 │ T       │ C       │
│   NULL │ 1       │  21850037 │ C       │ T       │
│   NULL │ 1       │ 111324762 │ G       │ T       │
│   NULL │ 1       │  21564093 │ C       │ T       │
│   NULL │ 1       │ 144439973 │ C       │ T       │
│   NULL │ 1       │  31432986 │ C       │ T       │
│   NULL │ 1       │  62475346 │ G       │ T       │
├────────┴─────────┴───────────┴─────────┴─────────┤
│ ? rows (>9999 rows, 20 shown)          5 columns │
└──────────────────────────────────────────────────┘

All in the chromosome 1, pretty odd...
Here is what my chorosome csv file looks like :
[t1adm_al5v@bi0uappp003 test]$ cat ../hg38.chrom.sizes.csv
1,248956422
10,133797422
11,135086622
12,133275309
13,114364328
14,107043718
15,101991189
16,90338345
17,83257441
18,80373285
19,58617616
2,242193529
20,64444167
21,46709983
22,50818468
3,198295559
4,190214555
5,181538259
6,170805979
7,159345973
8,145138636
9,138394717
MT,16569
X,156040895
Y,57227415

Strange can you send me a sample of your data ?

Here it is : the vcf and parquet files of one single sample.
I had to add the .txt suffixe to ba able to attached them, so you may have to remove the suffixes before looking at them...
2081345853.vcf.txt
2081345853.parquet.txt

I got it !
Except for the alternative chromosomes, which are not in my chromosome csv file, the only difference was that my file didn't have the chr,length header.
I cannot explain why but once I added the header variantplaner stopped creating Null variant_ids !
In short, it now works ! :)

Now, what is the smartest way to use this genotype sructuration by variants ?
How do you determine which id_part a variant belongs to ?
Thanks

I add an example in documentation https://natir.github.io/variantplaner/usage/#use-genotype-partition


Mapsui.Providers.Feature  indexer is case sensitive

The indexer for Mapsui.Providers.Feature uses a case sensitive dictionary with a string key.
private readonly Dictionary<string, object> _dictionary = new Dictionary<string, object>();
Running into problems attempts to match field names in shapefile dbf files.
If this dictionary was case insensitive, problem goes away:
Recommended fix:
private readonly Dictionary<string, object> _dictionary = new Dictionary<string, object>(StringComparer.InvariantCultureIgnoreCase);
https://github.com/garykindel/Mapsui/commit/d3bf392446faec2803d165560091c2ede3e8f968

Could you describe the situation in which this problem occurs? Is there use input involved?

In processing 1000+ shapefiles, I encountered several where field names returned by mapsui code (see below) did not match the field name string case in the dbf file itself.  Unfortunately, I lost track of the specific shapefiles with this issue.  If I can locate one of these files I'll share it.  Changing aforementioned dictionary to allow case insensitive string key fixed the issue.
Sample code to acquire feature field names.
        [TestMethod]         public void GetFieldNamesFromShapefile()         {             string shapefilePath = @"C:\DRC_Data\Arcview\CANADA\BC\BC_bedrocks_ll83.shp";             Assert.IsTrue(File.Exists(shapefilePath));             var wShapeFile = new Mapsui.Desktop.Shapefile.ShapeFile(shapefilePath);             var Boundingbox = wShapeFile.GetExtents();             var features = wShapeFile.GetFeaturesInView(Boundingbox, 1);             var wFeature= features.FirstOrDefault();             var wFieldNames = wFeature.Fields;             Assert.IsNotNull(wFieldNames);         }

Thanks for the sample code!
But how come Mapsui returns a different casing. Should the problem not be fixed at the source?

I thought about what would be the correct solution.  I like the idea of find fields without having to know case of the field name.  Some of the other database provider solutions for foxpro and dbf often force upper case on the field names because I think back in the day dbf had to have upper case field names.  My vote is change the dictionary.

I decided this to leave it as it was because I can see scenarios where this would be a problem and I don't want to introduce a breaking change here. I there still is a problem because of upper-casing then perhaps this could be solved in a different way.


Merge code coverage improvements

This is just pulling master into release/1.1.0 so that our code coverage improvements are integrated.

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/1.1.0@ab108b9). Click here to learn what that means.
The diff coverage is 86.44%.

@@               Coverage Diff                @@
##             release/1.1.0      #14   +/-   ##
================================================
  Coverage                 ?   93.14%
================================================
  Files                    ?       25
  Lines                    ?     1007
  Branches                 ?       46
================================================
  Hits                     ?      938
  Misses                   ?       67
  Partials                 ?        2

Impacted Files
Coverage Δ

...space/Service/Backend/BackendServiceProtocol.swift
100% <100%> (ø)

Tests/BackendServiceTests.swift
96.62% <100%> (ø)

...ests/Helper/Mocks/NetworkServiceMockSubclass.swift
100% <100%> (ø)

Tests/NetworkServiceTests.swift
100% <100%> (ø)

Tests/Helper/Mocks/MockBackendService.swift
33.33% <33.33%> (ø)

Tests/NetworkSessionTest.swift
85.71% <85.71%> (ø)

Tests/HTTPTest.swift
86.23% <86.23%> (ø)

Tests/NetworkRequestTests.swift
94.44% <90.32%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab108b9...a6a6ee8. Read the comment docs.


Implement file handling in the entrypoint

This PR will implement the file handling for the entrypoint

Thanks, nice PR


[v1.14] ipsec: Safely delete Xfrm state

[ ] #32450 @jschwinger233

Once this PR is merged, a GitHub action will update the labels of these PRs:
32450

/test-backport-1.14

/test-backport-1.14


[WebApp] simplify code structure

@xscript,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Codecov Report

Merging #48 into master will decrease coverage by 0.93%.
The diff coverage is 76.1%.

@@            Coverage Diff             @@
##           master      #48      +/-   ##
==========================================
- Coverage   80.81%   79.88%   -0.94%
==========================================
  Files          41       38       -3
  Lines        1220     1198      -22
  Branches      127      127
==========================================
- Hits          986      957      -29
- Misses        183      190       +7
  Partials       51       51

Impacted Files
Coverage Δ

...pp/handlers/PublicDockerHubRuntimeHandlerImpl.java
70% <ø> (-2.73%)
:arrow_down:

.../maven/webapp/handlers/JavaRuntimeHandlerImpl.java
64.28% <ø> (-2.39%)
:arrow_down:

.../maven/webapp/handlers/NullRuntimeHandlerImpl.java
100% <ø> (ø)
:arrow_up:

...pp/handlers/PrivateRegistryRuntimeHandlerImpl.java
50% <ø> (-2.39%)
:arrow_down:

...p/handlers/PrivateDockerHubRuntimeHandlerImpl.java
50% <ø> (-2.39%)
:arrow_down:

...a/com/microsoft/azure/maven/AbstractAzureMojo.java
55.12% <0%> (-6.31%)
:arrow_down:

...va/com/microsoft/azure/maven/function/RunMojo.java
94.91% <100%> (ø)
:arrow_up:

...com/microsoft/azure/maven/function/DeployMojo.java
83.72% <100%> (ø)
:arrow_up:

...zure/maven/webapp/handlers/HandlerFactoryImpl.java
68.42% <100%> (ø)
:arrow_up:

...a/com/microsoft/azure/maven/webapp/DeployMojo.java
76.59% <74.41%> (-23.41%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 089e0a2...fc93602. Read the comment docs.


reduced album image size

Examples:

Now looks much better

Also created icon for OP button. Can send you if you want to change it.

The new downloader is not yet complete which is why the image size is to big and the other changes are unnecessary.


Saving to DDS lost alpha channel

Trying upscale DDS texture with trensparency. On preview all ok. When saving to PNG - all ok. But when save to DDS alpha is absent (got black).

Did you turn on the transparency feature?

Yes. In preview and saving png it works.

I set output format to "same as source" and got the following message
Cupscale does not support the image format  for exporting, so PNG is used.


Use res.json to return JSON

A little bit cleaner

Ok, up to you! I just feel sending a JavaScript object to a method called json is clearer than setting headers and passing a JSON string, but it’s your repo 👍


surprising scores on short strings

Hi! We've been playing with tinyld for identifying the language of short search queries and have been a little surprised by strings that seem pretty clearly english to us being very hard for it to give us high accuracy signals. Is it a known limitation that tinyld struggles with short text?
"search sprint 1" gives us
Merge Results [
{ lang: 'ga', accuracy: 0.08333333333333333 },
{ lang: 'et', accuracy: 0.044066666666666664 },
{ lang: 'ro', accuracy: 0.03285 },
{ lang: 'es', accuracy: 0.030449999999999994 },
{ lang: 'en', accuracy: 0.014425000000000002 }
]
with only=en, we get an accuracy of 0.117 for english on that string
new hire onboarding, only=en -> 0.058
codebase modularization, only=en -> 0

Yes It's a normal problem, to avoid repeating myself I created a FAQ and answered here
I have few ideas that could help for shorter string accuracy but nothing magical

thanks for the link, apologies we hadn't found that already.
On Thu, Nov 10, 2022 at 12:15 AM Kevin Destrem @.***>
wrote:

Closed #19 https://github.com/komodojp/tinyld/issues/19 as completed.
—
Reply to this email directly, view it on GitHub
https://github.com/komodojp/tinyld/issues/19#event-7779696870, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADMZMBHUHAZRVPBK5SSGOTWHSAIHANCNFSM6AAAAAAR3ZW75U
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
David Blackman
creative technologist & wandering
help me find my purpose http://purpose.blackmad.com


how to filter on existing url

i am a beginner in django mptt urls....thanks in advance
class Category(MPTTModel):
title = models.CharField(max_length =120)
parent = TreeForeignKey('self' , null = True , blank = True , verbose_name='parent category',       related_name='categories')
def get_absolute_url(self):
return reverse('categories', kwargs={'path': self.get_path()})
def get_clothes(self):
return Product.objects.filter(category__in=self.get_descendants(include_self=True))
class Product(models.Model):
brand = ForeignKey('Brand', verbose_name='brands', related_name='brand' , default='')
category = TreeForeignKey('Category', verbose_name='categories', related_name='products' , default='')
i have categories like this
men > men clothing > tshirts
my template
{% for product in instance.get_clothes %}
{{ product.title }}
{% endfor %}
how can i filter based on brand and still show the result on the same template ?

Filtering by brand is something more complicated than just changing your template and models. You also need to do some work in views.py.
This would take time, and this question is not related to the sources of this repo. I cannot write your code for you. Learn django basics on djangoproject.com.
P.S.
You don't need TreeForeignKey in your Product model, and default='' in ForeignKey is also nonsense. Your related_name='brand' is ambiguous, make it more verbose and explanatory. Correct your model:
class Product(models.Model):
    brand = ForeignKey('Brand', verbose_name='brands', related_name='products')
    category = ForeignKey('Category', verbose_name='categories', related_name='products' )


module:make & namespace

Is there a step that I am maybe missing? Everything seems correctly installed, config published, and yet when I run
php artisan make:module Test
namespaces for generated files stay like this:
namespace {{path}}\and\the\rest\which\is\correct;
Seems like {{path}} is not being correctly replaced. At least in my instance. So, for example when I change format function in:
vendor/caffeinated/modules/src/Console/Generators/MakeModuleCommand.php
to:
	/**
	 * Replace placeholder text with correct values.
	 *
	 * @return string
	 */
	protected function formatContent($content)
	{
		return str_replace(
			['{{slug}}', '{{name}}', '{{namespace}}', '{{version}}', '{{description}}', '{{author}}', '{{license}}', '{{path}}'],
			[$this->container['slug'], $this->container['name'], $this->container['namespace'], $this->container['version'], $this->container['description'], $this->container['author'], $this->container['license'], $this->module->getNamespace()],
			$content
		);
	}

(notice added {{path}} replacement from default function) it starts making namespaces correctly. Not sure if someone already reported this or not. Did a quick scan of closed issues, didn't find it. Maybe didn't search hard enough. So, a bug possibly?  Actually might end up forking it anyway for our purposes, but that is another story. Thanks in advance.

Definitely a bug - thanks for reporting~ I'll have this patched shortly.
No closed issue on this, but there is an open one for the same issue found here: https://github.com/caffeinated/modules/issues/158


[Social Interactions] "Looking at me" all are labelled as false

Hi Ego4D team,
Thank you so much for releasing this benchmark! It's great!
I downloaded the annotation file yesterday. After taking a short scan of the data, I've noticed that for av_train.json and av_val.json, all "social_segments_looking" are labeled as false. It's a bit odd, since it should contain both true and false labels. Would you please help take a look at that and see if I'm missing anything?
Best,

Please ignore that.

Take a look at the benchmark. https://github.com/EGO4D/social-interactions/blob/lam/dataset/data_loader.py#L126 It seems to me all social_segments_looking "is_at_me" should be labelled as True. Not looking at me is not labelled.

Yes, that's correct.  We have an update coming out shortly, but you can essentially treat all looking at me annotations as True for practical purposes.

Awesome. Thanks for confirming that!


Ollama support added. With a test connecting to a phi-3 mini model

Ollama support code is missing? Unable to use abi_base parameter for LLM class


Clear network/model descriptions

We would like to have a human understandable description of each of the benchmark tests. Past of that should include a visual representation of the architecture/topology of the  neural network.
(From 5/17 meeting)

Not a rules issue.


Update snippets for<EMAIL_ADDRESS>

I also added newlines to all of the snippet files.

i'm so sorry i have to close this pull request. i haven't notice i'm not watching it and only i see this pull request and only now when i upgraded to v4 i see this.
thank you and i'm really sorry again :(

No worries! And thanks!


Syntax error when using webpack 5 and karma 6

Apologies if this isn't the correct place to report this, but getting a bit desparate. I've converted my app to work with webpack 5, but I'm unable to get the tests running. All configs look correct. I've also tried more or less plugging in the example config from here, but still getting the same error:
https://github.com/appzuka/karma-webpackv5-test
Getting a strong feeling that something related to webpack or karma is completely off, but I'm unable to figure out what it is.
As described below, the weird message I get no matter what I do is this:
Chrome 89.0.4389.114 (Windows 10) ERROR
  Uncaught SyntaxError: Unexpected token ':'
  at C:/Users/einarq/AppData/Local/Temp/_karma_webpack_1/commons.js:3:66

I've pasted the beginning of that generated file below.

Operating System: Windows 10
Node Version: 14.15.4
NPM Version: 6.14.10
webpack Version: 5.31.0
karma-webpack Version: 5.0.0

Expected Behavior
Tests should run fine
Actual Behavior
08 04 2021 20:38:17.964:INFO [karma-server]: Karma v6.3.2 server started at http://localhost:9999/
08 04 2021 20:38:17.965:INFO [launcher]: Launching browsers ChromeNoSandbox with concurrency unlimited
08 04 2021 20:38:17.971:INFO [launcher]: Starting browser Chrome
08 04 2021 20:38:20.213:INFO [Chrome 89.0.4389.114 (Windows 10)]: Connected on socket Arz-bQzd3v3C9CF0AAAB with id 68760601
Chrome 89.0.4389.114 (Windows 10) ERROR
Uncaught SyntaxError: Unexpected token ':'
at C:/Users/einarq/AppData/Local/Temp/_karma_webpack_1/commons.js:3:66
SyntaxError: Unexpected token ':'
Code
{
      mode: 'development',
      output: {
        publicPath: '/',
      },
      cache: true,
      devtool: false,
      resolve: {
        modules: [path.resolve('./www/js/app'), 'www/js/libs', 'www/svg', 'node_modules', 'test/jasmine-ui', 'server'],
        extensions: ['.js', '.jsx', '.html', '.css', '.svg'],
        fallback: {stream: require.resolve('stream-browserify')},
      },
      plugins: [
        new webpack.ProvidePlugin({
          process: 'process/browser',
        }),
      ],
      module: {
        rules: [
          {
            test: /\.(jsx|js)?$/,
            include: [path.resolve(__dirname, 'www/js/app'), path.resolve(__dirname, 'server')],
            use: [
              {
                loader: 'babel-loader',
                options: {
                  cacheDirectory: true,
                },
              },
            ],
          },
        ],
      },
    },
    webpackMiddleware: {
      noInfo: true
    },
  }

Geneated file contents:
{
/***/<EMAIL_ADDRESS>/*!*****************************************************************!*\
  !***<EMAIL_ADDRESS>***!
  \*****************************************************************/
/*! dynamic exports */
/*! export __esModule [maybe provided (runtime-defined)] [no usage info] [provision prevents renaming (no use info)] -><EMAIL_ADDRESS>.__esModule */
/*! other exports [maybe provided (runtime-defined)] [no usage info] -><EMAIL_ADDRESS>*/
/*! runtime requirements: module, __webpack_require__ */
/***/ ((module, __unused_webpack_exports, __webpack_require__) => {

module.exports = __webpack_require__(/*! core-js/library/fn/date/now */ "./node_modules/@babel/runtime-corejs2/node_modules/core-js/library/fn/date/now.js");

/***/ }),

/***/<EMAIL_ADDRESS>/*!**************************************************************************!*\
  !***<EMAIL_ADDRESS>***!

Related issue: https://github.com/ryanclark/karma-webpack/issues/505

This is a much smaller repro, I'll try and look at this sometime over the weekend to troubleshoot.

I found a workaround over in #505 . I'm still looking on how to make a proper fix though.

As mentioned in the other issue, deleting the "browserslist" solves the problem.
It also works if I simplify it to just specify this for example:
"last 2 Chrome versions"
I tried a some different approaches to overriding this in my karma config, but was not able to do so successfully.
Perhaps you know of a way to do this?
I tried this:
{
  loader: 'babel-loader',
  options: {
    cacheDirectory: true,
    presets: [['@babel/preset-env', {targets: 'last 2 Chrome versions'}]],
  },
}

Actually, I was able to get it working by putting this in my karma.conf.js:
module.exports = function(config) {
  process.env.BROWSERSLIST = 'Chrome >= 80, Firefox >= 80, IE >= 11';

Btw, part of the problem is probably that we use "split bundling" currently, to serve a modern bundle to those who are worthy :)
Contents of "browserlist" file:
[modern]
last 2 Chrome versions
not Chrome < 60
last 2 Firefox versions
not Firefox < 54

[legacy]
last 5 Chrome versions
last 5 Firefox versions
last 5 Edge versions
Safari >= 10
iOS >= 10.1
IE >= 11

https://github.com/babel/babel-loader/issues/902
Opened an issue over here to bring attention to this

I'm working on the same project as @einarq , and can it definitely looks to be something with the way our browserslist is defined.
If we remove the environments from our browserslist and f.ex change the browserslist to either:
last 2 Chrome versions
not Chrome < 60
last 2 Firefox versions
not Firefox < 54

or
last 5 Chrome versions
last 5 Firefox versions
last 5 Edge versions
Safari >= 10
iOS >= 10.1
IE >= 11

the tests runs fine.
Another option is to leave the browserslist as-is with both environments defined, and instead set the target config in karma/webpack config to either target: 'browserslist:modern or target: 'browserslist:legacy'. When doing this, the tests also run fine. This is what we are doing for now.

I am getting the same Unexpected token ':' error.
I have tried deleting the .browserslistrc file.
I have tried setting target: 'browserslist with only targeting a single version of Chrome.
No joy :(

@einarq In this issue, you describe working with karma 6. However, in the linked "example" repo you are using karma 5. Do you have an example "karma 6" repo?

@peaceful-james
Sorry, don't remember all the details, but I think the "fix" we did on our end was to add this to our karma config:
      target: 'browserslist:modern',

@haakemon Isn't that correct?

target: 'browserslist:modern',

What does your .browserslistrc file look like?

We have a file called simply "browserslist" (not a dot file, no extension),
and it looks like this:
[modern]
last 3 Chrome versions
not Chrome < 60
last 3 Firefox versions
not Firefox < 54
[legacy]
last 5 Chrome versions
last 5 Firefox versions
last 5 Edge versions
Safari >= 10
iOS >= 10.1
IE >= 11
On Wed, Aug 25, 2021 at 9:59 AM Peaceful James @.***>
wrote:

target: 'browserslist:modern',
What does your .browserslistrc file look like?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ryanclark/karma-webpack/issues/507#issuecomment-905273974,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABT6DZ27XVBNPIJKN7WEALT6SPGZANCNFSM42UNNVNA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

@einarq Thanks. I will try that. The target: 'browserslist:modern', that you put in karma.conf.js, is that a root level config option? I can't find it in the docs.

It's a webpack option, so we put it on the weppack part of the
karma.conf.js file:
webpack: {
mode: 'development',
output: {
publicPath: '/',
},
cache: true,
target: 'browserslist:modern',
On Wed, Aug 25, 2021 at 10:20 AM Peaceful James @.***>
wrote:

@einarq https://github.com/einarq Thanks. I will try that. The target:
'browserslist:modern', that you put in karma.conf.js, is that a root
level config option? I can't find it in the docs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ryanclark/karma-webpack/issues/507#issuecomment-905288965,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABT6D2LDOPIH5ZD3HSZYMLT6SRUTANCNFSM42UNNVNA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

I have done some more investigation and I think my problem is actually different in nature to the OP.
I have a lib called draft-js that has a particular line in one of the functions:
  process.env.NODE_ENV !== "production" ? warning(selection.isCollapsed(), 'moveSelectionBackward should only be called with a collapsed SelectionState') : void 0;

This is strange to me because the source on github (https://github.com/facebook/draft-js/blob/10ca1ad44843e970c4314f85a8f37d26f842ebf9/src/component/handlers/edit/commands/moveSelectionBackward.js#L27) just says:
warning(selection.isCollapsed(), 'moveSelectionBackward should only be called with a collapsed SelectionState')

So my node_modules code is different to the code in their repo. Anyway...
I get the syntax error on how this line is interpreted. In /var/folders/qb/xxxxxxxxxxxxxxxxxxxx/T/_karma_webpack_348757/commons.js I am seeing this:
{"key1":"value1","key2":"value2"}.env.NODE_ENV !== "production" ? warning(selection.isCollapsed(), 'moveSelectionBackward should only be called with a collapsed SelectionState') : void 0;

So the process is being converted into a huge dictionary/object/map (whatever it's called) representing every key-value pair in process.
I'm sorry if I have cluttered this issue with something unrelated but for posterity I'm leaving my discoveries.

Final note: I should not have written anything in this issue. My problem was caused by process not being defined, and me trying to define it in a hacky way. I have fixed my problem by using:
        new EnvironmentPlugin({
          NODE_ENV: 'development', // use 'development' unless process.env.NODE_ENV is defined
          DEBUG: false
        })

in my webpack "plugins" config. Sorry for all the noise 🎺

@peaceful-james I appreciate your diligence in reporting your findings, those posts may someday be useful to someone troubleshooting the same issue 😉


Curious if you test this on a non-git folder?

I'm just wondering if we should add error-handling or a try/catch wrapper.

I tested it. The promise rejects.
I wonder what would be the expected behaviour from a users perspective?
In case no commits have been performed in the project should the hash and tag be empty strings?

@pke Can you make it so that it only adds the properties (otherwise omits them) if the promise rejects?  So you would wrap try/catch here https://github.com/pke/parse-app-info/blob/master/index.js#L26-L29 I think.

I'll version bump and publish, one moment!


Blog App: Controllers specs

In this PR, I generated Rspec tests for requests.
Changes

Installed the  rails-controller-testing gem.
Populated view templates.
Generated request spec tests for the users and posts controllers.
Checked the response status for both controllers.
Checked whether the template was rendered for both controllers.
Checked if the response body includes the correct placeholder text for both controllers.

Thanks for the feedback, @cynthiainga. I'm on it.

Thanks, @Munsa1 for the feedback and for approving my PR. Have a wonderful day. 🙃


using is-string-blank module now

Now that there is a separated module for testing blank strings, this PR uses is-string-blank as a dependency. The performance of this module is tested on plotly.js branch on CircleCI here items numbered 17003-8.
This would resolve issue 2 .
@alexcjohnson

Beautiful, thanks! We don't have CI on this repo, but it does have a test suite (npm run test). Does it still pass?

Yes I've got 272 successes there.

💃


Remove version: TheDocumentFoundation.LibreOffice version <IP_ADDRESS>

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://download.documentfoundation.org/libreoffice/stable/24.2.4/win/x86/LibreOffice_24.2.4_Win_x86.msi
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 912)


getting "Null" when player tries to accept quest.

What is your Quests version / build number (do not say "latest")?
3.8.1
Describe the problem and how to replicate it. Any console errors to post using Pastebin.com?
[Quests] RiplyGamer attempted to take quest "null" but something went wrong
If applicable, can you provide an example quest from quests.yml using Pastebin.com?
https://pastebin.com/5DqFMJrU
Hey sorry to bug you again, I gave the general default player group all the non admin perms for questing but they still cannot accept the quest when prompted yes or no. I am able to do them though (as op and admin) but not players.

addition/edit: It's only happening for old man jenkin's gimme bacon quest. Elgrog's iron ore quest works (but that was the pre made quest) I'm still unsure what I did wrong.

I have not been able to replicate this in my test environment with the correct permissions, but will continue to investigate and update accordingly.

huh interesting o,o  the issue didn't occur to me when I did it but it did happen when a default player tried to do it though if that helps at all. :D

Duplicate of #993

@MrRayhonda Would you be able to provide the following information? Myself and another collaborator are still investigating this.

Operating system (Windows/Mac/Linux)
Major Java version (Java 8, Java 11, et al.)
Permissions plugin and version

@PikaMug Sure thing!
Windows 10
The java thing I'm not sure how to check. Do you mean on my pc that I have installed or on the server? The Server.jar is the Paper server for 1.14.4 (not entirely sure how to check any more info on that) and my PC java version is version 8 update 181
MC Version is 1.14.4
LuckPerms 4.4.1 for permissions

Thanks for that!
I meant on the server that you're running Quests on. On a Windows server, a great way to check for Java is to open a new Command Prompt and type java -version then hit Enter. It may show a lot of information, but all we need is the version number.

Np! Sadly the server isn't hosted on my computer. I have this pterodactyl back end thing I use that my friend set up to get the server hosted etc. I'm totally new to this and use my pc for work but wanted a 24 hour server that my community can use. I checked the setup parameters and see this "java -Xms6G -Xmx7G -jar server.jar" dunno if that helps at all

That's alright, thank you again for trying. The setup parameters you mentioned just tell the server how much memory to use (in your case, between 6 and 7 gigabytes). If you're using Pterodactly Panel (link) it's most likely that you're running Java 8.

Oh okay cool! I'm glad we could figure that out. That is what I'm using it seems. I didn't set it up but the screen shots match the website back end

I've tried several times to replicate this bug, but without any luck. Since the trail has gone cold, I'm closing this in order to focus effort on the original issue. Please direct further input to #993


Using a proxy for pop3

Hello,
The documentation doesn't say much about it, but is it possible to use proxies for accessing pop3 with the mail gem?
Thanks

I'm also interested in using this gem via a proxy -- is that possible?  thanks

I need this too! Great gem thanks!

so ? Proxy support ? where ? when ?

No proxy support. Consider a PR setting up a POP3Connection retriever method, similar to the SMTP and SMTPConnection retriever methods, that allows you to provide your own POP3 that's set up however you like.


PHOENIX-7140 Update Apache Parent and Maven Plugin Versions in Core

Apache Parent pom updated to v30
Versions for the following plugins removed as they are derived from parent pom:

maven-enforcer-plugin
maven-shade-plugin
maven-project-info-reports-plugin

Unused property surefire.version has been removed
Properties findbugs-annotations.version, maven-owasp-plugin.version, and os.maven.version have been relocated
Versions for these plugins are now managed via properties:

jasmine-maven-plugin
lifecycle-mapping
maven-bundle-plugin
exec-maven-plugin
maven-checkstyle-plugin

Upgraded versions for these plugins:

maven-eclipse-plugin to 2.10
maven-build-helper-plugin to 3.5.0
restrict-imports.enforcer to 2.4.0
spotbugs-maven-plugin to <IP_ADDRESS>
spotbugs to 4.8.1
jacoco-maven-plugin to 0.8.11
sonar-maven-plugin to <IP_ADDRESS>94
maven-owasp-plugin to 8.4.0
os.maven to 1.7.1
maven-bundle-plugin to 5.1.9
exec-maven-plugin to 3.1.1
maven-checkstyle-plugin to 3.3.0

Updated checker.xml based on updated checkstyle, refer https://github.com/checkstyle/checkstyle/issues/2116
Switched to  as the old declaration variant no longer supported, refer https://github.com/skuzzle/restrict-imports-enforcer-rule/releases/tag/v2.0.0
Temporarily retained old version of apache parent for maven-dependency-plugin as upgrading is causing build to fail

Ran following locally and verified build is SUCCESS:

mvn clean install -DskipTests
mvn clean compile test-compile site -Dspotbugs.site
mvn checkstyle:checkstyle
mvn pmd:pmd
mvn exec:exec -q -N -Dexec.executable="echo" -Dexec.args='${hbase.profile.list}'

CC: @stoty

You can just use git commit --amend, and edit the patch in your favourite unix-ish command line editor.
Maybe your editor uses CR instead of NL ?

Sorry, misclicked.

 Maybe your editor uses CR instead of NL ?

Phew! Fixed the message:
https://github.com/apache/phoenix/commit/47bf1371abe9ab43d200c1e436bdfdc9dae34030.patch Thank you.
Force pushed: updated DTD version to be same as omid, not sure how to validate as mvn checkstyle:checkstyle runs seemlessly locally without the change also, so waiting for build result.


Enable sherlodoc on .odoc-index file for odoc 3 compat

This adds support for the future odoc 3, which include generating .odoc-index files for sherlodoc to consume instead of consuming .odocl files directly.
It works with the following odoc PR : https://github.com/ocaml/odoc/pull/1084

Would it be possible to add a little test to generate an .odoc-index and a pin to the odoc PR in sherlodoc, such that the CI will run? Thanks! :)

The CI seems to report errors with the latest odoc CLI which I'm not familiar with... Any ideas on how to update the cram tests? :)

If I understand correctly the CI uses regular odoc, instead of the latest version from master. The corresponding PR in odoc is merged, so a pin on master should fix this.

I have updated this to work with the latest odoc
The CI still fails, but only on macos and fails with a weird message : error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)


Add monte_carlo_pi.jl

CPU/threading requires https://github.com/JuliaFolds/Transducers.jl/pull/433


test_member_CLUDS fails during cleanup in Mitaka

Title: test_member_CLUDS fails during cleanup in Mitaka
Labels: Bug
Milestone: Leave this field empty.
Assignee: @richbrowne
Attachments:

/etc/neutron/server.log server.log.txt

Agent Version
LBaaSv2 version is 8.0.2
agent version is 8.0.2
Operating System
CentOS 7
OpenStack Release
Mitaka
Description
While running the test_CLUDS in mitaka test_member_CLUDS fails during cleanup while trying to delete pools.  This was the fifth test to run. The interesting errors in the server.log file starts at 12:35:33. Have
discussed this problems with @jlongstaf and @zancas
Deployment
Session is still up: flanigan-over-mit-116-0526-01  : This is a overcloud setup in a mitaka version of openstack.  BigIP version 11.6

I need to see the f5-openstack-agent.log file too.  For some reason the
loadbalancer is still in UPDATE state, which means the previous operation
is not yet completed.
On Thu, May 26, 2016 at 2:18 PM, Dennis Flanigan<EMAIL_ADDRESS>wrote:

Title: test_member_CLUDS fails during cleanup in Mitaka
Labels: Bug
Milestone: Leave this field empty.
Assignee: @richbrowne https://github.com/richbrowne
Attachments:

/etc/neutron/server.log server.log.txt
https://github.com/F5Networks/f5-openstack-lbaasv2-driver/files/285233/server.log.txt

Agent Version
LBaaSv2 version is 8.0.2
agent version is 8.0.2
Operating System
CentOS 7
OpenStack Release
Mitaka
Description
While running the test_CLUDS in mitaka test_member_CLUDS fails during
cleanup while trying to delete pools. This was the fifth test to run. The
interesting errors in the server.log file starts at 12:35:33
Deployment
Session is still up: flanigan-over-mit-116-0526-01 : This is a overcloud
setup in a mitaka version of openstack. BigIP version 11.6
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/F5Networks/f5-openstack-lbaasv2-driver/issues/113

I looked at this but saw no errors in the f5-openstack-agent.log around the time that this test failed.

This issue is likely caused by the way in which the polling client is initialized and implemented.  In some cases the .4 second polling interval is too short and causes the api call to happen when the prior one is still in progress.  This results in an exception which is not handled by the polling client.
The call to delete should not be retried.  In fact, the proper way is to test the existence of the object, and if it exists try to delete, then poll for the existence until it is removed.
So the pattern is:
if x.exists()
x.delete()
while x.exists() and retries < max_retries:
sleep (interval)
if x.exists():
return ERROR


Support for pushing arbitrary byte sequences in cf_string channels

We would like to transmit arbitrary byte sequences using pylsl and cf_string channels. Unfortunately, the current pylsl implementation does not support that. Technically, it should work just fine though.
liblsl states the following about the cft_string channel format:
/** For variable-length ASCII strings or data blobs, such as video frames, complex event
      descriptions, etc. */
cft_string = 3,

For the user's convenience, pylsl cf_string outlets accept Python str objects and encode them on the fly. Similarly, the cf_string inlets decode incoming byte strings to str objects.
While this is helpful 99% of the time, it makes sending/receiving arbitrary byte strings impossible, e.g. when one wants to transmit encoded video frame data. Decoding our bytestring to utf-8 first and then have pylsl encode it again is unfortunately not a viable workaround as not all byte sequences can be decoded to utf-8.
Proposed solution
We suggest adding a boolean flag to the corresponding pull_* and push_* function that indicates wether the data should be encoded/decoded, defaulting to True. This should fit conceptionally, given that it is the users responsibility to interpret the data.
Dear pylsl maintainers, what do you think about this proposal?
/cc @ClaraKuper

Just thinking out loud...
We could add another pylsl-only channel_format for cf_bytestring. This would require an additional mapping from cf_bytestring to the same lib.pull_ and lib.push_ functions as cf_string, but it wouldn't pass the cf_string check to trigger the encode/decode.
How does that sound?
I prefer doing something like this instead of adding a kwarg that is only relevant in << 1% of cases and might be confusing if not documented exactly correctly.

@cboulay I think this is a very good alternative. 👍🏻 @clarakuper and I will work on an implementation proposal.

I heard from @ClaraKuper in Slack that you are now encountering a problem with your char-arrays being interpreted as null-terminated strings, effectively cutting your data whenever a 0 appears. This is indeed inconsistent with the doctoring.
There's another push_ method in lsl's C-API that can handle your use-case. Unfortunately it is not exposed in pylsl, probably because the method requires an additional argument so it would be hard to make the call transparent to the user.
https://github.com/sccn/liblsl/blob/05f7e16525930058c2e00681b10da9031fa7b90e/src/lsl_outlet_c.cpp#L161-L173
Since we are providing the length of the char array, it uses the 'from sequence' std::string constructor (5 here), and it shouldn't do a null termination.
So, as above, if you use a new cf_bytestring type, you can modify the function maps to have that entry redirect to lib.lsl_push_sample_buftp. I don't know how to best handle the extra argument; it might require passing a ..., *args.


UI: extend pipeline states to show local as well

Task
The pipeline states page only has remote now, but using the same mechanisms minus the api show local information as well

We need more info here, may have related tasks that also need to be done. Need general refactoring on the Pipeline States page.


Vagrant build rocket from downloaded git

From the commit:
Previously the vagrant configuration would grab the last release
of Rocket to build. Now Rocket will be built on whatever is in the
checked out git repository (/vagrant inside the VM)
This will allow testing of changes to Rocket using the "vagrant
provision" command.
It will also automatically use the latest rocket version.
Assumes ./build script handles cleanup.
Go and APPC versions still manually configured in provisioning scripts

lgtm, merging.


Update info on no-proxy property with wildcards

Added wildcard information for the no-proxy property. including the fact that some  jobs accept *. as wildcard and some others . to prefix domains with wildcards.
Perhaps we should recommend to type both in case the user is not sure? Example: always type *.foo.bar and .foo.bar if you want to have a wildcard for whatever.foo.bar
Signed-off-by: Greg Patricio<EMAIL_ADDRESS>

We updated the release notes with the bug fix for the wildcard domains. Please check to adapt it to the common style for release notes.

Thank you, @gregpatricio


Proxy Device fails to start

Required information

Distribution: Ubuntu
Distribution version: Ubuntu 18.04.2 LTS
The output of "lxc info" or if that fails:

Kernel version:
LXC version: 4.15.0-50-generic
LXD version: 3.13 (snap)
Storage backend in use: zfs

Issue description
A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.
Everything was working fine until after a reboot.
Now lxc doesn't open any ports and trying to modify results in an error:
$  lxc config device remove haproxy myport443 Error: Invalid devices: Not an IP address: localhost
Steps to reproduce

Add a proxy device
lxc config device add haproxy myport443 proxy listen=tcp:<IP_ADDRESS>:443 connect=tcp:localhost:443
Unsure. Worked fine for some time.
$  cat /var/snap/lxd/common/lxd/logs/haproxy/proxy.myport443.log Error: Not an IP address: localhost Error: Not an IP address: localhost

Information to attach

[ ] Any relevant kernel output (dmesg)
[x] Container log (lxc info NAME --show-log)
[x] Container configuration (lxc config show NAME --expanded)
[x] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[x] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

lxc_info.txt
container_configuration.txt
container_log.txt
daemon_debug.txt
daemon_log.txt
ltc.txt

Use lxc config edit to fix all errors in one shot?

@stgraber Thanks, so simple. I change all occurences of localhost to <IP_ADDRESS>. Seems to work in a first test. Going to change the rest.
I tried to change it with lxc config device set haproxy myport443 connect tcp:<IP_ADDRESS>:443  before which threw the same error.
Thanks for the fast help. Awesome.

Yeah, you had an invalid value that happened to work thanks to a bug in the proxy code, we fixed that bug and added the check. Changing a single line in the config causes it all to be re-validated which is why a simple add/remove would fail. edit lets you modify it all in one shot, making it pass validation.
Glad it worked and you're back to working order!


Command to find guest users will not find guests changed to member status

This command will find only users with userType = Guest, not members
Get-AzureADUser -Filter "userType eq 'Guest'"
This will find all users (not the first 100) and with a CreationType -eq "Invitation", all external users
get-azureaduser -all $true | ?{$_.CreationType -eq "Invitation"}
or using the new graph commands
Connect-MgGraph -scopes "directory.read.all"
Get-MgUser -All -Filter "CreationType eq 'Invitation'"  | select displayname, mail, id
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6e00be85-c1bd-bfd9-e09b-78fea811f864
Version Independent ID: bdfb2aad-6aa8-76a8-46cf-1603f8255df4
Content: Prevent guests from being added to a specific group
Content Source: microsoft-365/solutions/per-group-guest-access.md
Product: microsoft-365-enterprise
GitHub Login: @MikePlumleyMSFT
Microsoft Alias: mikeplum

@debaxtermsft Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/8526
Once the author approves, the changes will be live.
Thanks
Sri

@msbemba Thank you very much for the contribution and updating the documentation with PR. @debaxtermsft Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


TypeError in /clan/prwz-br-frz-2vjg9v2y

Error in ClashLeaders
TypeError in /clan/prwz-br-frz-2vjg9v2y
Failed to fetch
View on Bugsnag
Stacktrace
unknown file:0 - unknown method

View full stacktrace
Created automatically via Bugsnag

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/zhi-chi-tian-ya-hai-kuo-tian-kong-l99j8vvq

An error linked to this issue has been reopened in Bugsnag
TypeError in /

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/farm-with-us-ccqp0j2

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/parallax-2y09lv28

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/sound-of-fury-2vy0l9y0

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/parallax-2y09lv28

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/slovak-brothers-2rj2ycgr

An error linked to this issue has been reopened in Bugsnag
TypeError in /

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/clan-games-vl09cy8l

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/clan-games-vl09cy8l

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/zhong-guo-da-xue-sheng-bu-luo-rvprvlgg

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/full-attack-2u99c9ur

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /about

An error linked to this issue has been reopened in Bugsnag
TypeError in /

An error linked to this issue has been reopened in Bugsnag
TypeError in /

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/7th-heaven-8yuc0cpq

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/exit-humanity-200gljc2g

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/zhong-guo-da-xue-sheng-bu-luo-rvprvlgg

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/qing-tong-zai-shou-shui-neng-yi-jian-tu-long-9g28080v

An error linked to this issue has been reopened in Bugsnag
TypeError in /clan/wu-wang-chu-zhong-9pqpgru0


Add newly deployed cloud friend to roster

after cloudInstall we now:

sign into the Cloud social network if we aren't yet logged in
add isAdmin=true to the object returned by the installer
pass this networkData to acceptUserInivtation

cloudInstall also now just returns a Promise<void>, since the UI doesn't need to know the invite anymore.

This change is

Just made the other changes, PTAL, thanks!

Let's export that field! Then we'll be good to go. :+1:, pending that.

Damn! OK, :+1:.


sensor.hacs: repositories attribute failing

Version of HACS

Latest
Describe the bug
Since the introduction of the sensor.hacs attribute "repositories", it always report the value  "[object] [object]"  (when an update is present).

That is expected.

Only when you have some time, can you explain the attribute meaning or point me to a source to understand this?


Custom matching regex

If we could specify our own complete regex for matching it would make it more powerful.

Yes allowing wildcards like grunt-cache-bust would save a lot of extra configuration time. Eg.
match: [
    'app-components/**/*.js',
]


⚠ Auth has degraded performance

In 1920b60, Auth (https://auth.k2lin.com) experienced degraded performance:

HTTP code: 502
Response time: 2762 ms

Resolved: Auth performance has improved in 4283d3f.


chore: Release 1.0.1

:robot: I have created a release beep boop
1.0.1 (2024-07-08)
🐛 Bug Fixes

Make finding correct title work on Windows with CRLF (01e356c)

📦 Build System & Dependencies

Update dependencies (383396f)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v1.0.1
:sunflower:


Dlrm

breaks docs for some reason

@Borda for some reason this breaks the docs...

Codecov Report

Merging #95 into master will decrease coverage by 0.06%.
The diff coverage is 88.73%.

@@            Coverage Diff             @@
##           master      #95      +/-   ##
==========================================
- Coverage   92.66%   92.60%   -0.07%
==========================================
  Files          73       74       +1
  Lines        3708     3772      +64
==========================================
+ Hits         3436     3493      +57
- Misses        272      279       +7

Flag
Coverage Δ

#unittests
92.60% <88.73%> (-0.07%)
:arrow_down:

Impacted Files
Coverage Δ

pl_bolts/datamodules/sklearn_datamodule.py
88.88% <87.50%> (+0.13%)
:arrow_up:

pl_bolts/utils/dataset_download.py
88.70% <88.70%> (ø)

pl_bolts/datamodules/__init__.py
100.00% <100.00%> (ø)

pl_bolts/models/rl/dqn_model.py
83.78% <0.00%> (ø)

pl_bolts/models/rl/per_dqn_model.py
84.31% <0.00%> (ø)

pl_bolts/models/rl/noisy_dqn_model.py
75.00% <0.00%> (ø)

pl_bolts/models/rl/double_dqn_model.py
100.00% <0.00%> (ø)

pl_bolts/models/rl/n_step_dqn_model.py
100.00% <0.00%> (ø)

pl_bolts/models/rl/dueling_dqn_model.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bad440...b3d245a. Read the comment docs.

Borda for some reason this breaks the docs...

what does it break, do you mean the Conv section?


Review use of panic in datafusion-optimizer crate

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":111                 new_expr.pop().unwrap(),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":145                 predicate: new_expr.pop().unwrap().pop().unwrap(),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":167                 window_expr: new_expr.pop().unwrap(),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":189             let new_aggr_expr = new_expr.pop().unwrap();
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":190             let new_group_expr = new_expr.pop().unwrap();
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":212                 expr: new_expr.pop().unwrap(),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":276         let (expr, _, data_type) = expr_set.get(&id).unwrap();
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":557         unreachable!("Enter mark should paired with node number");
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":655         let (_, counter, _) = self.expr_set.get(curr_id).unwrap();
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/common_subexpr_eliminate.rs":676             || self.expr_set.get(id).unwrap().1 <= 1
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/filter_push_down.rs":180         _ => unreachable!("lr_is_preserved only valid for JOIN nodes"),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/filter_push_down.rs":195             JoinType::Semi | JoinType::Anti => unreachable!(
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/filter_push_down.rs":200             unreachable!("on_lr_is_preserved cannot be applied to CROSSJOIN nodes")
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/filter_push_down.rs":202         _ => unreachable!("on_lr_is_preserved only valid for JOIN nodes"),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/filter_push_down.rs":314             .chain(once(on_to_keep.into_iter().reduce(Expr::and).unwrap()))
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/simplify_expressions.rs":172         _ => panic!("Expected boolean literal, got {:?}", expr),
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/simplify_expressions.rs":393         if self.can_evaluate.pop().unwrap() {
"/home/andy/git/apache/arrow-datafusion/datafusion/optimizer/src/single_distinct_to_groupby.rs":176                             fields_set.insert(expr.name(input.schema()).unwrap());

Describe the solution you'd like
Review code that can panic and see where it makes sense to return a Result instead. For example, It is generally better to use ? than unwrapon results.
The goal is not to remove all panics but review and make sure we are using them appropriately. Bonus points for adding documentation for invariants.
Describe alternatives you've considered
None
Additional context
Generated by https://github.com/andygrove/no-need-to-panic

These have all been resolved now.


ja.json update

Github's diff doesn't show any changes. Is it possible that you've accidentally overwritten your changes during one of the merge commits?

Closing in favor of #139


Test of Deleting testResources via an internal branch & pullRequest

By merging this pull request, then the testResources file should be deleted from the Master.

Yes, that worked.


More advanced 3D rendering

🚀 Feature
Following up on https://github.com/napari/napari/pull/840#issuecomment-570379878 from @royerloic - right now for 3D arrays we have translucent, mip, additive, and iso surface rendering. @royerloic and @AhmetCanSolak mentioned another type of rendering that can be nicer than mip and that has one free parameter that you can use to control - i can't remember what it was though. If you could give the name / a link to the rendering equation then I can try implementing it as a new option, which should be easy now that we have a copy of the vispy volume shader code in our repo

@sofroniewn it's attenuated max projection. Loic talks about it here: https://youtu.be/tSMRoW0-NFY?t=830 It's probably useful for 3D picking as well!

Great video - that guy really looks like he knows what he's talking about - would be great if he could make a shader PR to napari ;-) ;-)
@royerloic do you have any links to the equations for this plot?

One approach I that could imagine is you sum how much you've seen along the ray so far and then scale by something inversely proportional to that maybe scale = exp(-lambda * sum(values_to_current_point)) so if nothing has been seen so far you have no scaling (scale=1) but then the more you see, the larger the sum, and the smaller the negative exponent. The lambda affects the severity of this scaling, so lambda = 0 is just a normal max intensity projection
This should be pretty easy to implement, so I'll give it a go now. Happy to try other schemes too! I have some cryoEM data that I'd like to make look pretty!!

hahaha...
Yes, I think it is due time now that I dwelve into some shader code....
and make a nice PR... But I am afraid, @sofroniewn is probably going to get
this done by the time I finish to write this...
For reference, here is the good ol' ClearVolume shader code:
https://github.com/ClearVolume/clearvolume/blob/master/src/java/clearvolume/renderer/opencl/kernels/VolumeRender.cl
The interesting bit is here:

In fact, the really interesting bit is this:

Essentially what you described @sofroniewn
Which can be 'ported' to the more complex shader infrastructure of vispy, and added to the actual volume shader code...

One detail is to scale the decay_rate so that the effect does not overall depend on the number the length of the ray-marching step along the ray (tstep in the code)

Thanks @royerloic - I've given this a go in #846, still a little to do, but already looking quite nice I think!
I think you have some offsets / do the mapping slightly differently than I did. I'm happy to adjust etc as needed, the basic effect is very cool though.


Set the popper components dynamic z-index

Description

How to config that (in docs i found nothing about this)?
How to config for example z-index for Notice component global?

Validations

[X] Ensure this issue neither a bug report nor a feature proposal.
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that descript the same thing to avoid creating a duplicate.

Can use the install options or ConfigProvide to specify a zIndex initial value. (This part is missing in document, sry)
Can specify a z-index style for .vxp-notice. All Notice items are under a wrapper that has .vxp-notice class.

For best practise, I need wrap inside one ConfigProvider all poppers or wrap every popper singly?

If you use app.use to install, you can do like:
import { install } from 'vexip-ui'

app.use(install, {
  zIndex: 12345
})

If you not use app.use, I recommend you wrap app inside a ConfigProvider to instead of ths above.

Well, I‘’m developing a hook function for better usage. Use like:
<script setup lang="ts">
import { provideConfig, configProps } from 'vexip-ui'

// For multiple
provideConfig({
  props: { size: 'large' }
  // locale, icons, zIndex
})

// For single
configProps({ size: 'large' })
</script>


Extend HintRules with fields for GHC parse trees

Another step towards conversion of Match.hs to GHC parse trees.
This PR extends type HintRule with fields hintRuleGhcLHS, hintRuleGhcRHS, hintRuleGhcSide and populates them with parse trees in Yaml.parseRule.
Right now, Grep.runGrep and other functions in Translate that create HintRules values by direct construction of abstract syntax fill those fields with dummy values. It appears, that'll have to be fixed next.

Oh dear. We've got a bit more of
hlint: Stack space overflow: current size 33624 bytes.
hlint: Use `+RTS -Ksize -RTS' to increase it.
travis: Failed when running system command: hlint test +RTS -K1K

going on :-/

Following queues from https://github.com/ndmitchell/hlint/pull/694 and issue https://github.com/ndmitchell/hlint/issues/695, upped the stack limit to 4k in Travis.hs.

Looks awesome- CI and then I'll merge.

Looks awesome- CI and then I'll merge.

Bit stuck here. I've pushed the stack to 8k but we are still seeing Travis failures.
hlint: Stack space overflow: current size 33624 bytes.
hlint: Use `+RTS -Ksize -RTS' to increase it.
travis: Failed when running system command: hlint test +RTS -K8K

Looks awesome- CI and then I'll merge.
[...]
Update : Last ditch attempt with -K16k.

No joy 😞

Reported stack size in the error is probably wrong : https://gitlab.haskell.org/ghc/ghc/issues/10445. Reproduced locally, overcome with 512k. Pushed.


Rename marriage-abroad services

Trello card: https://trello.com/c/qMeYVdfh
This standardises the names of marriage-abroad services based on the information in the spreadsheet attached to the Trello card.
I'm hoping to get this merged and then update @erikse's PR #2404 to extract rates from the template into a data file.
Expected changes
These changes affect a lot of outcomes. The following list contains examples of each of the changes introduced in the commits in this PR:
Austria

Before and after
Replace: "Issuing a CNI or equivalent" with "Issuing a CNI, Nulla Osta or equivalent"

Albania

Before and after
Remove: "Issuing any other consular letter or certificate in English"
Remove: "Issuing any other consular letter or certificate in any other language"
Remove: "Administering an oath or making a declaration"

Cambodia

Before and after.
Reword: "Registering a marriage or civil partnership" to "Registering a civil partnership or marriage"
Reword: "Issuing a marriage or civil partnership certificate" to "Issuing a civil partnership or marriage certificate"

Colombia

Before and after.
Replace: "Posting a marriage notice" with "Receiving a notice of registration"
Replace: "Performing a marriage ceremony" with "Registering a marriage"

Qatar

Before and after.
Replace: "Affidavit for marriage" with "Affidavit or affirmation for marriage"
Remove: "Issuing any other consular letter or certificate in English"
Remove: "Issuing any other consular letter or certificate in any other language"

Finland

Before and after.
Replace: "Affirmation for marriage" with "Affidavit or affirmation for marriage"

Lebanon

Before and after.
Replace: "Affirmation for freedom to marry in English" with "Affidavit or affirmation for marriage"
Remove: "Affirmation for freedom to marry in any other language"

China

Before and after.
Replace: "Affirmation or affidavit for freedom to marry in English & Chinese" with "Affidavit or affirmation for marriage"

Peru

Before and after.
Replace: "Affirmation or affidavit for freedom to marry" with "Affidavit or affirmation for marriage"

Egypt

Before and after.
Replace: "Marriage declaration" with "Affidavit or affirmation for marriage"
Remove: "Making an oath"

Hi @chrisroos great piece of work.. I found your very preview links useful.. I observed that the fees tables for Belarus, Japan and Oman appear empty.. is this intentional ??

Thanks for reviewing this, @ikennaokpala. And thanks for picking up on the empty fees tables! I'll investigate those and work out what we should do about them.

I'm currently working on an alternative to this branch. I'm extracting all the service/fee data from the templates into a yaml file. I think this is going to make it easier to both rename the services, and update the services offered by the various countries (both required by 6 April). I think it's also going to make the templates simpler.
I'm going to close this PR and will have the new one open by Tuesday.

The alternative approach to updating these services is in PR #2438.


initialOverallState does not transition to true when there are no registered checks

In the new health check functionality introduced in Dropwizard 2.1.0, if you set:
health:
  initialOverallState: false

but have no checks registered for a specific type (either alive or health), then the overall state will never transition to true.
This is because the overall app state is calculated by HealthCheckManager only when there is a state change from a health check:
https://github.com/dropwizard/dropwizard/blob/dd25ab7b61052721af2ebec54a6b8da5ced30edd/dropwizard-health/src/main/java/io/dropwizard/health/HealthCheckManager.java#L161
https://github.com/dropwizard/dropwizard/blob/dd25ab7b61052721af2ebec54a6b8da5ced30edd/dropwizard-health/src/main/java/io/dropwizard/health/HealthCheckManager.java#L197
And if there aren't any checks, then no state change triggers and the state is never recalculated (even though the counter of failed checks is actually zero).
I guess that Dropwizard must do an explicit test and in case there aren't any checks registered, consider that the condition is satisfied (technically it is) and instantly set the respective flag (isAppAlive or isAppHealthy) to true.

Hi @oneadr-3773. Thanks for reporting this.
I'm not completely convinced this is a general issue. The initialOverallState flag only sets the isAppHealthy field and not the isAppAlive field. Therefore the application should be reported as alive, but not as ready. And as only checks with type ready are affected by this, I would suggest not to set the initialOverallState to false, if you don't have a health check for this type in your application :wink:
The framework has no restiction on when to add a health check. Maybe one would want to spawn a service from within the application and register the health check afterwards. Then it wouldn't be correct to set the application's health state to true before the health check gets registered. For this case the current behavior is correct and necessary.

Hi. I understand your explanation, and I do think it makes sense, however I think there are some subtleties worth discussing. I believe that depending on how you interpret the setting you'll have a specific mental model and then the implemented behaviour may be surprising.
I understand that there are no restrictions on when to add a health check. However, the framework also can't know if and/or when a new health check will be registered. Therefore you could have a mental model in which initialOverallState means false until every (ready) registered health check at the point in time when run() finishes had the chance to run at least once (i.e., do not optimistically assume the healthy state).
For example, imagine you do have two health checks, A and B [1], registered early during startup. What the current implementation will do if you set initialOverallState to false is to stay false until both health checks run at least once. This is completely fine and expected. But note that the framework can't know that the code at runtime will register a third health check, C, in some unspecified point in the future. Obviously it won't "wait" for C to show up and be healthy in order to transition to the initial state to true, as that is impossible and no sane developer can have that expectation.
Therefore I do think that if you have a similar case, where you don't have any health check, then the framework should not "wait" for an unspecified future health check that may or may not come, it is consistent with the previous case for the framework just immediatelly transition to true as it waited for all health checks that it knew about once the run() method finishes (which would be the natural "checkpoint" given that the servlet just becomes exposed on that point). In this particular case, as there is no health check, then there is no failing health check, so no need to the app health to remain false once run() finishes. Naturally, if a new health check does end up being registered in the future, then the overall health state will transition accordingly.
I do understand that is not how's implemented, and I do think the current implementation makes sense under the circumstances you have explained, but to be honest I'm not convinced that will be the most likely mental model that developers will have in mind when they read the documentation. To me a much less surprising mental model would be:

initialOverallState is the assumed state of the health of the app until all health checks registered at the point in time when the run() method finishes execution become healthy.

I feel that this behaviour would be the one that is most aligned with developers' expectations. As an anedoctal data point, my team was experimenting with the new health functionality and encountered this behaviour accidentaly and we were all unanimously convinced that it was a bug -- and it was what motivated me to post here.
I understand that we are basically discussing two edge cases and arguing which one is more likely to happen without data to back it up. I also understand that API design is not an exact science, and a decision must be made, and once made and shipped it's not easy to change it. That said, if you are convinced that the current implementation is the least surprising then I would suggest that the existing documentation for this area could be improved [2]. In particular, it should explicitly say something like:

initialOverallState is the assumed state of the health of the app until there is at least one health check registered and all registered health checks become healthy. Note that if no health check is ever registered, then the state will never transition to true and in this case it should be left in its default (true) state.

To conclude, I hope that I am bringing some interesting new perspective and useful suggestions. Maybe this is already something you have considered during API design and made an informed trade-off about (or maybe not). Personally, as an user of your project I would have made a different trade-off (and so my colleagues, I believe) but that's the nature of API design. Either way, let me know what your thoughts are.
[1] All health checks are of type ready for the sake of discussion.
[2] It's completely understandable, this being a totally new feature, that the documentation can lag behind a bit. IMHO the new health functionality is a bit under-documented right now, and some details you can only figure out by experimentation and/or inspecting Dropwizard's source code.

@oneadr-3773 Thanks for the detailed explanation. As our README states, the framework provides opinionated code, so this basically is what you said.

I do understand that is not how's implemented, and I do think the current implementation makes sense under the circumstances you have explained, but to be honest I'm not convinced that will be the most likely mental model that developers will have in mind when they read the documentation.

The problem thereby is that we shipped with this implementation and that your suggestion would now be incompatible to that. But we could think of changing this with the 3.0.0 release. As I'm not a core developer of the dropwizard-health module, I'd refer to @natalie-zamani and @pstackle for that.
Anyway, the documentation should be up-to-date. As you mentioned, the current documentation doesn't fully explain the initialOverallState, so we would be happy to receive a PR on that.

Totally understood. My main goal was to bring this to your attention. I agree that changing the behaviour of a shipped version must meet a very high bar and this certainly does not qualify, but maybe it's something you would consider tweaking in a future major release.


Add API Doc for Schema Registry Version 2022-10 API

Data Plane API - Pull Request

The Schema Registry service from Azure Event Hubs now supports storing Json and custom schemas. This new API doc describes the additions to the API endpoints to support client-side manipulation of schema registry resources.
API Info: The Basics
Most of the information about your service should be captured in the issue that serves as your engagement record.

Link to engagement record issue:

Is this review for (select one):

[ ] a private preview
[x] a public preview
[ ] GA release

Change Scope
This section will help us focus on the specific parts of your API that are new or have been modified. Please share a link to the design document for the new APIs, a link to the previous Open API document (swagger) if applicable, and the root paths that have been updated.

Design Document:
Previous Open API Doc: Previous Doc
Updated paths: "specification/schemaregistry/data-plane/Microsoft.EventHub/stable"

❔Got questions? Need additional info?? We are here to help!

 Contact us!
The Azure API Review Board is dedicated to helping you create amazing APIs. You can read about our mission and learn more about our process on our wiki.

💬 Teams Channel
💌 email

Click here for links to tools, specs, guidelines & other good stuff
Tooling

Open API validation tools were run on this PR. Go here to see how to fix errors
Spectral Linting
Open API Hub

Guidelines & Specifications

Azure REST API Guidelines
OpenAPI Style Guidelines
Azure Breaking Change Policy

Helpful Links

Azure DevTools Wiki

Leaving comments about content type of custom and Json schemas open; if service side changes are needed, will resolve comments here when those changes are complete.


os: StartProcess clearing O_NONBLOCK can lead to goroutine hang on Linux

What version of Go are you using (go version)?

go version go1.15.7 linux/amd64

Does this issue reproduce with the latest release?
Yes
What did you do?
func TestHang(t *testing.T) {
	c, err := net.Dial("udp", "<IP_ADDRESS>:0")
	if err != nil {
		t.Fatal(err)
	}
	defer c.Close()

	f, err := c.(*net.UDPConn).File()
	if err != nil {
		t.Fatal(err)
	}
	defer f.Close()

	cat, err := os.StartProcess("/bin/cat", []string{"/bin/cat"}, &os.ProcAttr{
		Files: []*os.File{os.Stdin, os.Stdout, os.Stderr, f},
	})
	if err != nil {
		t.Fatal(err)
	}
	defer cat.Wait()
	defer cat.Kill()

	go func() {
		time.Sleep(time.Second)
		t.Log("Closing")
		c.Close()
	}()

	buf := make([]byte, 1)
	// This read never returns.
	t.Log(c.Read(buf))
}

What did you expect to see?
The read on c returns after the goroutine calls Close(). The same problem exists for other syscalls like accept.
What did you see instead?
The read on c blocks indefinitely. The root cause is that os.StartProcess calls f.Fd() which in turn calls poll.FD.SetBlocking. That method does the following:

Clear O_NONBLOCK from the open file descriptor flags
Set fd.isBlocking to true

The problem in the example I've given is that O_NONBLOCK is shared by both f and c, but only f.isBlocking is ever set to true. The call to c.Close() sees that c.isBlocking == 0 and therefore blocks while trying to acquire runtime_Semacquire(&fd.csema) in poll.FD.Close. The same interaction is possible when using exec.Cmd.ExtraFiles since that just ends up calling os.StartProcess.
I think there are two related problems here:

Clearing O_NONBLOCK from an fd can hang the go runtime.
StartProcess clears O_NONBLOCK.

This behaviour is the root cause of a bug I fixed my library: https://github.com/cloudflare/tableflip/commit/cae714b289e199db5da5f08af861ea65be6232c0 I was aware that the runtime clears O_NONBLOCK when calling File.Fd() while writing the library, and tried my best to avoid the problem and still got stung by this.

CC @ianlancetaylor; see also #29277

CC @ianlancetaylor; see also #29277

This is basically a dup of #29277.  We pretty much have to set the file to blocking mode when we pass the descriptor to the child process.  Many processes will fail if they are given a non-blocking standard input.

This is basically a dup of #29277.  We pretty much have to set the file to blocking mode when we pass the descriptor to the child process.  Many processes will fail if they are given a non-blocking standard input.

Yeah, I agree. The only additional thing here is that the os.StartProcess behaviour is not documented anywhere, one has to go look at the source as far as I can tell.
I understand the reasoning for stdin, stdout, stderr, but should the same apply to other files as well? It seems like touching the file status flags is best avoided as much as possible.

Yeah, I agree. The only additional thing here is that the os.StartProcess behaviour is not documented anywhere, one has to go look at the source as far as I can tell.
I understand the reasoning for stdin, stdout, stderr, but should the same apply to other files as well? It seems like touching the file status flags is best avoided as much as possible.

Should this be closed in favor of #29277, or is it helpful to track this issue separately?
I'll move this to Backlog as it seems this won't be fixed in 1.17.

I think the only thing to do here beyond #29277 is a doc fix.  Sending a CL.


Formatting of "az aks addon enable" makes it difficult to read.

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 26e8d0d1-269d-f2af-840e-f935a7c84aa5
Version Independent ID: ffbd9d63-da65-259a-9ac3-7750d3783f83
Content: az aks addon
Content Source: latest/docs-ref-autogen/aks/addon.yml
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

A few new lines would help this

route to service team

@olsenme to look into to.

Can you please explain the problem a little more?

Wanted to follow up to see if this is still an issue?


Add Extension Localization to Spanish, French, and German

Added Localization to de (German), en (English), en_GB (English Great Britain), en_US (English United States), es (Spanish), es_419 (Spanish Latin America and Caribbean), and fr (French)


Create an 'objectives' page/ folder for PLN & Dev Tools

Describe the use case
We need each page to be consistent and jump to an 'objectives' page when you open that section from the curriculum page
Describe the problem
Each page jumps to the second, 'introductions page' when you open a section from the curriculum page
Proposed Solution
Create an objectives page for both:

[ ]  PLN
[ ] Dev Tools
So that there exists a landing page for each of those sections
[ ] Update layouts/index.html ln:23 to say 'objectives instead of `introductions

PR is here: https://github.com/protocol/launchpad/pull/367

Dependent on https://github.com/protocol/launchpad/pull/371

Completed


How to download more than 2000 images?

Hello, i want to ask. How to change the limit of 2000 images download ? I have almost 500 member that i follow on pixiv, but the software just download 2000 images from my following list. How to remove the 2000 images limit?  Thanks before, i'll wait for your answer.

Get a pixiv premium.
Non-premium member only allowed to get up to 2000 search result from the
new images pages.
Else, you need to get all the member id (use the export function) and then
use the download from list function.

Get a pixiv premium. Non-premium member only allowed to get up to 2000 search result from the new images pages. Else, you need to get all the member id (use the export function) and then use the download from list function.

Alright,  gonna do the export function then. Thanks!

Can you tell me how to do it ? Do i need just put all of the number id to the list.txt ?

Alright thank you.


Unable to parse IP addresses in hostname part of user info

The library doesn't handle IP addresses as the host in a message prefix.
Quote from the RFC
 servername =  hostname
  host       =  hostname / hostaddr
  hostname   =  shortname *( "." shortname )
  shortname  =  ( letter / digit ) *( letter / digit / "-" )
                *( letter / digit )
                  ; as specified in RFC 1123 [HNAME]
  hostaddr   =  ip4addr / ip6addr
  ip4addr    =  1*3digit "." 1*3digit "." 1*3digit "." 1*3digit
  ip6addr    =  1*hexdigit 7( ":" 1*hexdigit )
  ip6addr    =/ "0:0:0:0:0:" ( "0" / "FFFF" ) ":" ip4addr

Example message that can't be parsed.
<EMAIL_ADDRESS>JOIN :#test

Should be pretty easy to fix. I'll look into it. In general I think this parser should have more options in terms of customizing what is allowed and disallowed, but I am not an expert on IRC server/client behavior, so I don't know exactly what options would be relevant.

An additional parsing error on nicknames, they don't have to start with an alpha:
nickname   =  ( letter / special ) *8( letter / digit / special / "-" )

I noticed this user quit recently (IP address removed)
<EMAIL_ADDRESS>
Additionally (at least on Freenode and any other server that uses inspircd) a nickname can start with a digit. This occurs after a netsplit. When nicks collide, one of the nicks is reassigned to its network ID, which is always prefixed with a number as a matter of namespacing!

(Sorry for the flood, but it's all on the same topic)
isSpecial is missing the pipe character |
  special    =  %x5B-60 / %x7B-7D
                   ; "[", "]", "\", "`", "_", "^", "{", "|", "}"

I was using https://tools.ietf.org/html/rfc1459 as a guideline for parsing criteria.
<message>  ::= [':' <prefix> <SPACE> ] <command> <params> <crlf>
<prefix>   ::= <servername> | <nick> [ '!' <user> ] [ '@' <host> ]
<command>  ::= <letter> { <letter> } | <number> <number> <number>
<SPACE>    ::= ' ' { ' ' }
<params>   ::= <SPACE> [ ':' <trailing> | <middle> <params> ]

<middle>   ::= <Any *non-empty* sequence of octets not including SPACE
               or NUL or CR or LF, the first of which may not be ':'>
<trailing> ::= <Any, possibly *empty*, sequence of octets not including
                 NUL or CR or LF>

<crlf>     ::= CR LF

<target>     ::= <to> [ "," <target> ]
<to>         ::= <channel> | <user> '@' <servername> | <nick> | <mask>
<channel>    ::= ('#' | '&') <chstring>
<servername> ::= <host>
<host>       ::= see RFC 952 [DNS:4] for details on allowed hostnames
<nick>       ::= <letter> { <letter> | <number> | <special> }
<mask>       ::= ('#' | '$') <chstring>
<chstring>   ::= <any 8bit code except SPACE, BELL, NUL, CR, LF and
                 comma (',')>

Which RFC are you referring to? Perhaps this one is obsolete now.
Regardless, it would appear the RFC isn't followed very well, so there's definitely a strong case to be made for relaxing the parser. I think towards this goal, it would be a good idea to set up a record that allows you to specify valid tokens for different segments of a message.
I've consider a number of different formats for this record, and I think the best choice is to simply allow attoparsec parsers to be used to describe the tokens.
    data IRCParseConfig = IRCParseConfig {
        nickParser :: Parser ByteString,
        userParser :: Parser ByteString,
        ...
    }

This is nice in a number of ways. You can easily enforce the RFC definitions this way, and you can provide an extensive library of parsers to compose together and plug-in. The only issues I can see is that some user-defined parsers may become difficult to debug, but that's something that can be addressed later and isn't a major concern to me.

Do you have a short example log from an inspircd server I could use in a test case? The names can be mangled for privacy's sake. I'd just like a few canonical examples of IRC lines that use non-RFC formats.

I've been quoting from
https://tools.ietf.org/html/rfc2812
I'll have to put together a dump when I get back to this tonight.
On Mon, Feb 23, 2015, 12:35 PM Adam Curtis<EMAIL_ADDRESS>wrote:

Do you have a short example log from an inspircd server I could use in a
test case? The names can be mangled for privacy's sake. I'd just like a few
canonical examples of IRC lines that use non-RFC formats.
—
Reply to this email directly or view it on GitHub
https://github.com/kallisti-dev/irc-bytestring/issues/2#issuecomment-75626261
.

Alright so I've made a substantial number of changes to the package. I've added a configuration record as well as 2 modules for RFC1459 and RFC2812 parsing options. The only thing left is to add a "relaxed" parser and run some tests.


Add partition size parameter to SparkPersistor

Overview
This PR exposes the partition size as a parameter for the persist_wicker_dataset in the spark plugin instead of a hard-coded SPARK_PARTITION_SIZE const. To make the interface backward compatible, the parameter is set by default to the same value as the original const value. We also rename the const as SPARK_PARTITION_SIZE to DEFAULT_SPARK_PARTITION_SIZE to reflect the change.
Notes

This is backwards compatible because the partition_size parameter has a default value.

Testing

CI

Thanks @zhenyu , could you please bump up version (https://github.com/woven-planet/wicker/blob/main/VERSION) before merging?

Thanks @aalavian for reminder! done

LGTM @zhenyu! Thanks for the change, readable PR, and good description of the rationale. 🙇🏻♂

Thanks for the approval:)!