Saki Zenkoku Hen [PCSG00646]

Game summary

Game name: Saki Zenkoku Hen
Game serial: PCSG00646
Game version: 1.03

Vita3K summary

Version: v0.1
Build number: 1063
Commit hash: https://github.com/vita3k/vita3k/commit/deff8367

Test environment summary

Tested by: Zangetsu38
OS: Windows 10
CPU: AMD Ryzen 2700X
GPU: NVIDIA GTX 1080 Ti
RAM: 16 GB DDR4 3600 mhz

Issues

Crash on generate shaders.

Log
vita3k.log

Vita3K summary

Version: v0.1
Build number: 1075
Commit hash: https://github.com/vita3k/vita3k/commit/6142543

Issues

Spam infinite "OpenGL error: 0x502" on black screen
News shader error

[shader::usse::USSETranslatorVisitor::vpck]: Source not loaded

Log
vita3k.log

Vita3K v0.1.8 3205-9e20f6d0
still doesn't boot

as of https://github.com/Vita3K/Vita3K/commit/da8a6933d87478ca3425e27d47736a409b1bf85f, the game goes ingame (however i have a weak CPU, so i believe its playable at best).
Screenshots

Log
vita3k.log
Recommended Labels

Playable


Descriptive statistics

This PR introduces the following measurements/statistics:
numerical columns (float and int):

nan values count
nan values percentage
min
max
mean
median
std
histogram:

for float: fixed number of bins (which is a global config parameter - tell me if it's an overkill :D)
for integers: bin size is always an integer value. if difference between max and min is less then fixed number of bins parameter, bin size is equal to 1, otherwise it's round((max-min)/num_bins) - which means that bins might be not equal (like the last one being smaller) but otherwise there could be float bin edges which is nonsense.

categorical columns
These are only ClassLabel columns which is not ideal because some integers also might be categories, as well as strings, but we can't be sure in advance.

nan values count
nan values percentage
number of unique values
counts for each value

Here's the result for my toy dataset with 1000 rows

polinaeterna/delays_nans
{
    "num_examples": 100000,
    "statistics": [
        {
            "column_name": "class_col",
            "column_type": "class_label",
            "column_statistics": {
                "nan_count": 0,
                "nan_proportion": 0.0,
                "n_unique": 5,
                "frequencies": {
                    "this": 19834,
                    "are": 20159,
                    "random": 20109,
                    "words": 20172,
                    "test": 19726
                }
            }
        },
        {
            "column_name": "class_col_nans",
            "column_type": "class_label",
            "column_statistics": {
                "nan_count": 49972,
                "nan_proportion": 0.49972,
                "n_unique": 5,
                "frequencies": {
                    "this": 9904,
                    "are": 10021,
                    "random": 10126,
                    "words": 10061,
                    "test": 9916
                }
            }
        },
        {
            "column_name": "delay",
            "column_type": "float",
            "column_statistics": {
                "nan_count": 0,
                "nan_proportion": 0.0,
                "min": -10.206,
                "max": 8.48053,
                "mean": 2.10174,
                "median": 3.4012,
                "std": 3.12487,
                "histogram": {
                    "hist": [
                        2,
                        34,
                        256,
                        15198,
                        9037,
                        2342,
                        12743,
                        45114,
                        14904,
                        370
                    ],
                    "bin_edges": [
                        -10.206,
                        -8.33734,
                        -6.46869,
                        -4.60004,
                        -2.73139,
                        -0.86273,
                        1.00592,
                        2.87457,
                        4.74322,
                        6.61188,
                        8.48053
                    ]
                }
            }
        },
        {
            "column_name": "delay_nans",
            "column_type": "float",
            "column_statistics": {
                "nan_count": 49892,
                "nan_proportion": 0.49892,
                "min": -10.206,
                "max": 8.48053,
                "mean": 2.11288,
                "median": 3.4012,
                "std": 3.11722,
                "histogram": {
                    "hist": [
                        1,
                        17,
                        137,
                        7515,
                        4522,
                        1197,
                        6481,
                        22593,
                        7473,
                        172
                    ],
                    "bin_edges": [
                        -10.206,
                        -8.33734,
                        -6.46869,
                        -4.60004,
                        -2.73139,
                        -0.86273,
                        1.00592,
                        2.87457,
                        4.74322,
                        6.61188,
                        8.48053
                    ]
                }
            }
        },
        {
            "column_name": "temp",
            "column_type": "float",
            "column_statistics": {
                "nan_count": 0,
                "nan_proportion": 0.0,
                "min": 0.6,
                "max": 14.9,
                "mean": 7.28953,
                "median": 7.5,
                "std": 3.05441,
                "histogram": {
                    "hist": [
                        4711,
                        9781,
                        10349,
                        9781,
                        20166,
                        17046,
                        13965,
                        8794,
                        4055,
                        1352
                    ],
                    "bin_edges": [
                        0.6,
                        2.03,
                        3.46,
                        4.89,
                        6.32,
                        7.75,
                        9.18,
                        10.61,
                        12.04,
                        13.47,
                        14.9
                    ]
                }
            }
        },
        {
            "column_name": "temp_nans",
            "column_type": "float",
            "column_statistics": {
                "nan_count": 49959,
                "nan_proportion": 0.49959,
                "min": 0.6,
                "max": 14.9,
                "mean": 7.29404,
                "median": 7.5,
                "std": 3.05107,
                "histogram": {
                    "hist": [
                        2392,
                        4855,
                        5143,
                        4879,
                        10040,
                        8571,
                        7058,
                        4452,
                        2014,
                        637
                    ],
                    "bin_edges": [
                        0.6,
                        2.03,
                        3.46,
                        4.89,
                        6.32,
                        7.75,
                        9.18,
                        10.61,
                        12.04,
                        13.47,
                        14.9
                    ]
                }
            }
        },
        {
            "column_name": "vehicle_type",
            "column_type": "int",
            "column_statistics": {
                "nan_count": 0,
                "nan_proportion": 0.0,
                "min": 0,
                "max": 2,
                "mean": 0.82646,
                "median": 1.0,
                "std": 0.72333,
                "histogram": {
                    "hist": [
                        36343,
                        44668,
                        18989
                    ],
                    "bin_edges": [
                        0,
                        1,
                        2,
                        2
                    ]
                }
            }
        },
        {
            "column_name": "vehicle_type_nans",
            "column_type": "int",
            "column_statistics": {
                "nan_count": 50247,
                "nan_proportion": 0.50247,
                "min": 0,
                "max": 2,
                "mean": 0.82542,
                "median": 1.0,
                "std": 0.72384,
                "histogram": {
                    "hist": [
                        18135,
                        22169,
                        9449
                    ],
                    "bin_edges": [
                        0,
                        1,
                        2,
                        2
                    ]
                }
            }
        },
        {
            "column_name": "weekday",
            "column_type": "int",
            "column_statistics": {
                "nan_count": 0,
                "nan_proportion": 0.0,
                "min": 0,
                "max": 6,
                "mean": 3.08063,
                "median": 3.0,
                "std": 1.90347,
                "histogram": {
                    "hist": [
                        10282,
                        15416,
                        15291,
                        15201,
                        15586,
                        15226,
                        12998
                    ],
                    "bin_edges": [
                        0,
                        1,
                        2,
                        3,
                        4,
                        5,
                        6,
                        6
                    ]
                }
            }
        },
        {
            "column_name": "weekday_nans",
            "column_type": "int",
            "column_statistics": {
                "nan_count": 50065,
                "nan_proportion": 0.50065,
                "min": 0,
                "max": 6,
                "mean": 3.07762,
                "median": 3.0,
                "std": 1.90272,
                "histogram": {
                    "hist": [
                        5136,
                        7695,
                        7711,
                        7550,
                        7749,
                        7646,
                        6448
                    ],
                    "bin_edges": [
                        0,
                        1,
                        2,
                        3,
                        4,
                        5,
                        6,
                        6
                    ]
                }
            }
        }
    ]
}

Codecov Report
Patch coverage has no change and project coverage change: -0.96 :warning:

Comparison is base (4f40efa) 86.80% compared to head (7ee12fd) 85.85%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1199      +/-   ##
==========================================
- Coverage   86.80%   85.85%   -0.96%
==========================================
  Files          21       72      +51
  Lines         773     2622    +1849
==========================================
+ Hits          671     2251    +1580
- Misses        102      371     +269

Flag
Coverage Δ

jobs_cache_maintenance
99.08% <ø> (?)

jobs_mongodb_migration
82.65% <ø> (?)

services_admin
?

services_api
87.17% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

services/api/src/api/config.py
100.00% <ø> (ø)

... and 92 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

we could also use duckdb (should be faster and scale well) but afaik it doesn't implement numeric_histogram

@severo @lhoestq @AndreaFrancis can you please take a look at the current implementation? It's not as fast as I was hoping to for bigger datasets unfortunately but I believe it's because I didn't cook duckdb well :D and it might be improved.
The main problem here is that statistics for all features are computed in a single job. For bigger datasets that might take a long time but if it breaks in the middle for some feature, all the results for previously computed features will be lost (basically the same problem as with dataset-level parquet-and-info and Spawning urls scan). Maybe not a big deal for now when we set the limit? But we should take into account for future. (An obvious solution is to add another - feature - level to workers but not sure it's a good idea to add more and more granularity levels.)
Not marking the PR as ready yet but want to know your opinion.

@lhoestq for numerical stuff, but there are also more steps (three queries vs one query for categories now). I just checked for
amazon_us_reviews, Apparel_v1_00 config - it's 6M rows, about 1Gb parquet files size:
2023-06-16 13:42:17,790 INFO root Compute descriptive statistics for dataset='polinaeterna/amazon_us_reviews', config='Apparel_v1_00', split='train'
2023-06-16 13:42:17,791 INFO root Compute statistics for categorical features
2023-06-16 13:42:17,797 DEBUG root Compute statistics for ClassLabel feature **vine**
2023-06-16 13:43:25,743 DEBUG root ClassLabel **vine** computed
2023-06-16 13:43:25,743 DEBUG root Compute statistics for ClassLabel feature **verified_purchase**
2023-06-16 13:45:07,228 DEBUG root ClassLabel **verified_purchase** computed
100%|██████████| 2/2 [02:49<00:00, 84.72s/it]
2023-06-16 13:45:07,228 INFO root Compute min, max, mean, median, std, histogram for numerical features.
2023-06-16 13:45:07,228 DEBUG root Compute min, max ... for **star_rating**
2023-06-16 13:46:36,710 DEBUG root min, max ... for **star_rating** computed
2023-06-16 13:46:36,710 DEBUG root Compute nans for **star_rating**
2023-06-16 13:47:43,033 DEBUG root Nans for **star_rating** computed
2023-06-16 13:47:43,033 DEBUG root Compute histogram for **star_rating**
2023-06-16 13:48:49,270 DEBUG root histogram for **star_rating** computed

and then I broke because of HTTP error :( and I encounter this pretty frequently actually and wasn't able to debug it for a long time until the last bugfix release and this fix - it wasn't throwing an underlying error

@lhoestq @AndreaFrancis @severo can you please take a look? I've left comments where I'm not sure what to do. Basically I'm struggling with avoiding sql injections, hope you can advice what to do

Sure! Looking at it.

Great @polinaeterna, really nice PR!  I put a couple of minor/non critical comments

re

for float: fixed number of bins (which is a global config parameter - tell me if it's an overkill :D)

Perfect for now. There is plenty of room for configuration if we want, see https://observablehq.com/plot/transforms/bin for example:

These are only ClassLabel columns which is not ideal because some integers also might be categories, as well as strings, but we can't be sure in advance.

also taking example from Observable: https://observablehq.com/@observablehq/data-table-cell. Strings are considered as categories.

An obvious solution is to add another - feature - level to workers but not sure it's a good idea to add more and more granularity levels

what is your opinion after two weeks? I think it would make sense... a column, in columnar data, is a well-defined concept.
Another comment, related to the columns too: it would be good to have docs about what we support (column type, eg: we have issues with datetime) and which statistics we run (description of the stats, format)

@lhoestq i can't figure this out either, first I thought maybe because statistics takes too much time but for the same dataset with token it works :thinking:

Thanks docker
#0 30.32   • Installing types-pillow (<IP_ADDRESS>)
#0 30.80   • Installing types-psutil (<IP_ADDRESS>)
#0 31.03   • Installing types-requests (<IP_ADDRESS>)
#0 31.15   • Installing wget (3.2)
#0 38.14
#0 38.14   AssertionError
#0 38.14
#0 38.14
#0 38.14
#0 38.14   at /usr/local/lib/python3.9/site-packages/poetry/installation/executor.py:742 in _download_link
#0 38.16       738│         # to the original archive.
#0 38.16       739│         archive = self._chef.get_cached_archive_for_link(link, strict=False)
#0 38.16       740│         # 'archive' can at this point never be None. Since we previously downloaded
#0 38.16       741│         # an archive, we now should have something cached that we can use here
#0 38.16     → 742│         assert archive is not None
#0 38.16       743│
#0 38.16       744│         if archive.suffix != ".whl":
#0 38.16       745│             message = (
#0 38.16       746│                 f"  • {self.get_operation_message(operation)}:"
#0 38.16
#0 65.25   • Installing libcommon (0.6.8 /src/libs/libcommon)
------
failed to solve: executor failed running [/bin/sh -c poetry install --no-cache]: exit code: 1
make[1]: *** [up] Error 17
make: *** [test] Error 2`

I have the same error on main so probably just an environment error on my side - can someone else try to run the e2e tests locally ?

@severo @lhoestq @AndreaFrancis is the tests are passed now, can we merge?

Yes, let's merge!  (we can add the e2e tests later, I'll take care of it)

🎉🎉🎉 GOOD JOB!


Error Android when compile :app:mergeExtDexDebug

With this gradle:
./gradlew --version

Gradle 7.4.2
Build time:   2022-03-31 15:25:29 UTC
Revision:     540473b8118064efcc264694cbcaa4b677f61041
Kotlin:       1.5.31
Groovy:       3.0.9
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          17.0.6 (JetBrains s.r.o. 17.0.6+0-17.0.6b829.9-10027231)
OS:           Mac OS X 14.1.1 aarch64
Flutter doctor -v
[✓] Flutter (Channel stable, 3.13.5, on macOS 14.1.1 23B81 darwin-arm64, locale es-419)
• Flutter version 3.13.5 on channel stable at /Users/richarcangui/development/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 12fccda598 (3 months ago), 2023-09-19 13:56:11 -0700
• Engine revision bd986c5ed2
• Dart version 3.1.2
• DevTools version 2.25.0
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
• Android SDK at /Users/richarcangui/Library/Android/sdk
• Platform android-34, build-tools 33.0.0
• Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 15A507
• CocoaPods version 1.14.2
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2022.3)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.5)
• IntelliJ at /Applications/IntelliJ IDEA.app
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
[✓] VS Code (version 1.84.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.78.0
[✓] Connected device (5 available)
• LM X525 (mobile)          • LMX525USUKZP5D9PE6                   • android-arm    • Android 10 (API 29)
• iPhone de Richar (mobile) • 00008120-00044C500C00C01E            • ios            • iOS 17.1.2 21B101
• iPhone 15 Pro (mobile)    • 4084559E-4A21-450A-A93B-60D7184A85BF • ios            •
com.apple.CoreSimulator.SimRuntime.iOS-17-0 (simulator)
• macOS (desktop)           • macos                                • darwin-arm64   • macOS 14.1.1 23B81 darwin-arm64
• Chrome (web)              • chrome                               • web-javascript • Google Chrome 119.0.6045.199
[✓] Network resources
• All expected network resources are available.
• No issues found!
Show me this error when compile, maybe have any idea to solve this issue?
persona_flutter: ^3.1.2

What went wrong:
Execution failed for task ':app:mergeExtDexDebug'.

Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
Failed to transform inquiry-internal-2.10.6.aar (com.withpersona.sdk2:inquiry-internal:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/98b0af604352965770b1ba6c8537699e/transformed/jetified-inquiry-internal-2.10.6-runtime.jar.
Error while dexing.
Failed to transform dynamic-feature-2.10.6.aar (com.withpersona.sdk2:dynamic-feature:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.libraryelements=aar, org.gradle.status=release, org.gradle.usage=java-runtime}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/82fe16d367aef41784b3cbe2d1d0f83d/transformed/jetified-dynamic-feature-2.10.6-runtime.jar.
Error while dexing.
Failed to transform ui-2.10.6.aar (com.withpersona.sdk2:ui:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/8212f7a71a29d7639d8aba021cbe7302/transformed/jetified-ui-2.10.6-runtime.jar.
Error while dexing.
Failed to transform government-id-2.10.6.aar (com.withpersona.sdk2:government-id:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/5d23a90d142b1ca1837fe4dc4372dfbb/transformed/jetified-government-id-2.10.6-runtime.jar.
Error while dexing.
Failed to transform selfie-2.10.6.aar (com.withpersona.sdk2:selfie:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/5153765995fc6e74a00184039a7ac557/transformed/jetified-selfie-2.10.6-runtime.jar.
Error while dexing.
Failed to transform document-2.10.6.aar (com.withpersona.sdk2:document:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/24b2dceb5b7676a752ef410a004239fc/transformed/jetified-document-2.10.6-runtime.jar.
Error while dexing.
Failed to transform permissions-2.10.6.aar (com.withpersona.sdk2:permissions:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/7336ce77c6ac4fbfe5be27daecf2e53e/transformed/jetified-permissions-2.10.6-runtime.jar.
Error while dexing.
Failed to transform modal-2.10.6.aar (com.withpersona.sdk2:modal:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/56a726dd3942350672ea95568b73816b/transformed/jetified-modal-2.10.6-runtime.jar.
Error while dexing.
Failed to transform nfc-2.10.6.aar (com.withpersona.sdk2:nfc:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/e12c6f0dbabf0ad0d11b384888574337/transformed/jetified-nfc-2.10.6-runtime.jar.
Error while dexing.
Failed to transform ui-step-renderer-2.10.6.aar (com.withpersona.sdk2:ui-step-renderer:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/a48208b2cded583fde87a1ea9c4a4654/transformed/jetified-ui-step-renderer-2.10.6-runtime.jar.
Error while dexing.
Failed to transform network-2.10.6.aar (com.withpersona.sdk2:network:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/18324e65a2c8f1fbfd391ef6e9cdcf96/transformed/jetified-network-2.10.6-runtime.jar.
Error while dexing.
Failed to transform camera-2.10.6.aar (com.withpersona.sdk2:camera:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/fac8b9b89802459e7202da8996d90f15/transformed/jetified-camera-2.10.6-runtime.jar.
Error while dexing.
Failed to transform inquiry-types-2.10.6.aar (com.withpersona.sdk2:inquiry-types:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/8658381712aa7b71c2f8cd226fd2797b/transformed/jetified-inquiry-types-2.10.6-runtime.jar.
Error while dexing.
Failed to transform launchers-2.10.6.aar (com.withpersona.sdk2:launchers:2.10.6) to match attributes {artifactType=android-dex, asm-transformed-variant=NONE, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=24, org.gradle.category=library, org.gradle.status=release, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=androidJvm}.
Execution failed for DexingNoClasspathTransform: /Users/richarcangui/.gradle/caches/transforms-3/68fb5a84f1e77909edbd428696496977/transformed/jetified-launchers-2.10.6-runtime.jar.
Error while dexing.

Looks like some one was able to resolve this here
https://github.com/jorgefspereira/persona_flutter/issues/33


An inquiry channel can get stuck.

An inquiry was opened, but the prompt to close it stopped appearing in the tickets.

Closing until there is more evidence/cases of this having happened.


Bright tile 26226 observed with high read noise

Bright tile 26266 observed on 20220318 over three different exposures was reported by the observers to have high read noise on all of them.

The redshifts as a function of FIBERID look patchier than usual. They seem to have a pattern that changes petal by petal.

We discussed this on the 20220321 survey-ops call and decided that we would call these exposures bad (126715, 126716, 126717, 126718).  This is the three exposures on the BRIGHT tile (026226) and the one exposure on the following BACKUP tile (41779).  I'm not convinced that the read noise is actually an issue on this BRIGHT tile in very bright sky conditions, and I'm confident that it's not an issue on the following BACKUP tile, but for homogeneity's sake we'll take the very marginal loss of effective time.
@akremin, can you mark these as bad and reprocess?

@akremin , this one has somewhat high priority because it's one of the few tiles remaining in the early, southern part of the NGC, which is now setting.  It would be good to mark this exposure bad so that we can get more observations to complete this section of the footprint in Y1.

Apologies -- I missed this the first time around and am just getting to it after the ping yesterday.
The four exposures (126715, 126716, 126717, 126718) have been marked bad, and the spectra+redshift files associated with those exposures have been removed from daily. There were no other exposures for those tiles, so nothing needed to be rerun. I also don't see any later observations of the tiles, so no downstream cumulatives need to be redone either.
I just reran the tsnr script and the nigthly_qa script to reflect the removal of these exposures.
Back to you, Eddie.

Thanks, I cleared the QA here.


Invalid zip code in Campaign Edit page causes internal server error - correct workflow

Extension of issue #470 to make a new workflow fix and not just remove the error and for that matter we need to get on top of whole geo stuff anyways
To Start:
I've seen this same issue in the Create Campaign flow also. I guess this needs some discussion on how this should function.

Should it be an autocomplete from the available post codes in the PostalCodeGeo table?
What if someone is adding a campaign and their postcode doesn't exist yet?
How is the PostalCodeGeo anticipated to be managed to add/remove them?

@tonysurma, is there a reason we can't make PostalCode a required field when creating a Campaign? If the reason is the PostalCodeGeo lookup will fail, then my question to that is, what is PostCodeGeo table used for? Is it to generate this map?

If that's the case I guess a couple things:

disregarding our current lookup failure on PostCodeGeo, does it make sense for a Campaign to be created with no postal code? Is we don't require this up front, then this results in potentially a lot of checks on the read side for string.IsNullOrEmpty checks, etc... to avoid this, it's easier to require the postal code when we create a Campaign
taking PostCodeGeo table into account, would it be possible to show some text that says something along the lines of "to see this Campaign's Location on a map, please click here to fille out x, y, z". (or anywhere else on the UI which relies on the data in PostalCodeGeo to be available to see a map). Basically, something that allows us to still create a Campaign w/out postal code, but we end up hiding any UI elements that depend on PostalCodeGeo table to have matching data for that postal code


2.0.3 Minimum Stability?

  Problem 1
    - The requested package particlebits/pdo (installed at 2.0.2, required as 2.0.3) is satisfiable by particlebits/pdo[2.0.2] but these conflict with your requirements or minimum-stability.

Nothing changed in the requires in your lib, so I'm not sure why this error is being thrown. Can you confirm on your end?
When I set the requirement to ~2.0 in composer.json, the lock file is written as below:
             "name": "particlebits/pdo",
              "version": "2.0.2",

@mookman288 Could you try ^2.0 in your composer file and update? Mine is:
    "require": {
        "php": ">=7.0",
        "ext-pdo": "*",
...
        "particlebits/pdo": "^2.0",
    },

If that works I'll update the docs here. I'm not sure where that error is coming from though.

Are you able to try just putting in "2.0.3" and running composer update from a 2.0.2 install?
I attempted ^2.0, and it still only locks 2.0.2.

https://getcomposer.org/doc/articles/versions.md#exact-version-constraint
Exact version constraint needs to be supported. It's required for almost all of my projects, because I need to test every update of a third party dependency.
If it pulls down the most recent for you, it must be on my end, but I did test on three different environments. I'll give it another go with your suggestion above.

Yea let me clarify, I meant you should be able to get it to work with "^2.0" while we're testing, because if "^2.0" works then "2.0.3" will work. I think this specific error is cache related.

Sorry I didn't mean to sound demanding, I just meant that it's part of Composer.
Clearing the cache and removing the vendor directory did not work. Here's the snippet I found from running my project from a fresh install (using ^2.0, and 2.0.3) -
Package operations: 22 installs, 0 updates, 0 removals
  - Installing slim/slim (2.6.3): Downloading (100%)
  - Installing particlebits/pdo (2.0.2): Downloading (100%)

(I know 2.6.3 is old, but it's stable right now.)

Hi,
I just want to update that this might be related to some type of cache, either on github's end or on my end (despite testing in a few different environments.) After a few days, I haven't encountered this issue again.


error with h5ads from AnnData version 0.8.0

If the h5ad file has been made with most recent AnnData version 0.8.0, then it won't run in the pipeline. I tried to do this and got the error:
KeyError: 'dict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/iolosquires/sci-spatial-data/bin/router.py", line 20, in
fire.Fire(main)
File "/usr/local/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/Users/iolosquires/sci-spatial-data/bin/router.py", line 13, in main
out_file = h5ad_to_zarr(file=file_path, stem=stem, **args)
File "/Users/iolosquires/sci-spatial-data/bin/process_h5ad.py", line 22, in h5ad_to_zarr
adata = ad.read(file)
File "/usr/local/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 421, in read_h5ad
d[k] = read_attribute(f[k])
File "/usr/local/lib/python3.9/functools.py", line 888, in wrapper
return dispatch(args[0].class)(*args, **kw)
File "/usr/local/lib/python3.9/site-packages/anndata/_io/utils.py", line 183, in func_wrapper
raise AnnDataReadError(
anndata._io.utils.AnnDataReadError: Above error raised while reading key '/layers' of type <class 'h5py._hl.group.Group'> from /.
I think the pipeline requires the h5ad file to have been made using anndata version 0.7.8

Thanks @iolosquires , indeed it requires 0.7.8 as stated in https://github.com/haniffalab/sci-spatial-data/blob/dev/docker/Dockerfile.router#L10.
May I ask how the h5ad was generated? from spaceranger or ?

Hi Tong, sorry for my slow response. I believe it is from space ranger. If I open the h5ad. file and do "data.obs.keys" then each spot_id begins "space ranger..."

Np! - Ok, I see. Thanks for the info! Actually the pipeline now can also load raw space ranger output and convert to h5ads as well (so that we can be sure about anndata version). But that's not thoroughly tested. I'll add the docs for that once it's done.

@dannda closed c2362c88f0f00eba519bb259970689f5f98f5410


AddrN scheme

Zverik, hello
I am now in Georgia and opened a hellish problem - inability to map addresses of some buildings due to their position. It is corner buildings with two addresses. It would be cool to have a possibility to add both addresses through the addrN scheme. I know, it is a proposal feature, but current situation with constant data changing is a bit of conflict

Perhaps it might be best to add two entrance nodes, one on each side, and each with its own address ?
Or, one might help bring that AddrN proposal to fruition (or perhaps in discussion even better solution will emerge?)
As an alternative workaround (if you want to add those only-proposed features in every_door anyway) you can use raw tag editor to add any key=value pair as you want.

Hello @mnalis
About nodes with only addresses: as for me, I don't like this approach due to questions to final implementation, like mobile apps.
About proposals: main problem - it is a niche problem. Mostly, it is all about Eastern Europe. And we can argue heatedly for nothing. I am not in the sphere of mapping, I am neither geologist nor statistic. Not a civil engineer. I don't know how it usually works, best practice, etc. For me, map is map. Now, I can't suggest a final solution, alas
About manual entering key=value, yes, unfortunately, for now. Not so effective, but...

About nodes with only addresses: as for me, I don't like this approach due to questions to final implementation, like mobile apps.

Can you clarify what is the (perceived?) issue with mobile apps you are having with that approach (mapping multiple addresses on same building as separate entrance=* nodes with addr:*=*?)
All the ones I use (OsmAnd, Organic maps, StreetComplete, Every Door, Acastus Photon, Vespucci, etc.) have no problem with addresses being mapped as separate nodes instead of on building polygons. (In fact, I would be really surprised if any OSM apps had problems with them, as they are used in about the same proportion as addresses on polygons, so any app using only addresses on building would miss addresses on half of the world!)
Mapping separate entrances with multiple addresses is also the (more precise) approach that EveryDoor supports - click on the :house: (house) icon, and then  on the :door: (door) icon to add it. It will snap the address on the correct building.

About proposals [...] we can argue heatedly for nothing. I am not in the sphere of mapping, I am neither geologist nor statistic. Not a civil engineer.

I think Eastern Europe is somewhat bigger than a niche, though. Also practically nobody doing the proposals is geologist or statistics or civil engineers. Just people trying to get something done on OSM. But it does require certain patience and willingness to listen to comments made by others (trying to nullify or ignore what others are saying is what can make it heated :smile:), so I understand people might find it taxing.

I don't know how it usually works, best practice

Neither does anybody else - that is actually the whole point of proposal process - to collect information from various individuals with wildly different use cases; and then try to make a solution that would work for majority of them.

Can you clarify what is the (perceived?) issue with mobile apps you are having with that approach

It is subjective, but as for me, node upon a building doesn't show a relation to the building at all. It's just a node. And just a building. It would have a meaning if these two entities were in a relation (as a data type). But they aren't.

it does require certain patience and willingness to listen to comments made by others (trying to nullify or ignore what others are saying is what can make it heated 😄)

In IT, we do not accept toxicity. Only direct aggression, yea :)

tagging firstly needs to be accepted by community

take a knife and look at another mapper on the other side of the street (joke)
But in total, you are right about non-standardized approaches. Basic things create more and more problems when you begin to disassembling them

It is subjective, but as for me, node upon a building doesn't show a relation to the building at all. It's just a node. And just a building. It would have a meaning if these two entities were in a relation (as a data type). But they aren't.

Hmm, but they are quite related! Not as a OSM relation data type, but in the same manner as relation consists of nodes, ways, and other relations -- it is the same with way: it consist of many nodes.
Here is random example around here:
https://www.openstreetmap.org/way/976743183 is a building polygon made up of many nodes. Some of nodes that comprise that way are different address nodes, e.g. https://www.openstreetmap.org/node/9038628353 or https://www.openstreetmap.org/node/9038628354
So those nodes are a part of that way - you can't get a stronger connection that they are "related" than that! (even relation OSM data type is arguably weaker). So it is absolutely clear to any data consumer that wants to know that those addresses are part of that exact building polygon.
Or are you thinking of cases where nodes aren't connected to the building polygon, but just with coordinates close it it? I agree that such situation might pose a problem with relating them to the building, but:

mobile apps that you mention just don't care (they care where that address is, not part of what building it is. In fact, they prefer if the address is a node as it is more precise -- having an address on building can be problematic if building is huge and might be approached from several different streets)
Every Door "door" functionality will make sure  that the entrance is actually part of the building polygon.

Or are you thinking of cases where nodes aren't connected to the building polygon, but just with coordinates close it it

Yes, I mean exact that case.

So those nodes are a part of that way - you can't get a stronger connection that they are "related" than that! (even relation OSM data type is arguably weaker).

I thought that I understood duck typing earlier. I was wrong :))

@mnalis

Mapping separate entrances with multiple addresses is also the (more precise)
In some countries all addresses apply to the whole building, so putting several entrance nodes with different addresses is wrong. That's the reason why AddrN is a standard de-facto with country-specific approvals and, as always in OSM, nobody cares that some wiki pages states it as draft.

That's the reason why AddrN is a standard de-facto with country-specific approvals and,

I'd advise someone who uses that schema and cares about it to go and document its actual usage on the wiki. The fact that its status is "in use" or "de facto" instead of "approved" is (as you correctly note) of lesser importance. But its usage should at least be properly documented, if one expects that data consumers (like editors) should support it.

as always in OSM, nobody cares that some wiki pages states it as draft.

Well, you may not care, sure. But it you compare which data consumers use addr:housenumber vs. e.g. addr2:housenumber, you'll notice that "nobody cares" is likely not really correct.


What would it take to use oj 3.x?

oj 3 is pretty mature at this point, and offers advantages for rails projects.

(it's possible there is no issue, and the rails_51.gemspec is just out of date, as using gem "oj" with gem "rollbar" does not pin oj, we'll see if I have any reporting issues running rollbar in production)

Hey @lastobelus - Can you try this out as you described in your last comment and reopen the issue if you run into problems?


Discussing error management

There is an ongoing work of changing "error flags" to "error values". "error values" are primary values such as "null", "number", but they exist only on Jerry API level. Therefore JS do not need handle them at all.
Error values are designed to be return values, not arguments (no change here). Hence any API or C callback function can return with an error value, but the returned error value cannot be passed to other functions except a few error management functions and jerry_release. The value can be extracted from the error by jerry_get_error_from_value. The jerry_create_error_from_value creates an error value using the non-error value argument. For aborts, there will be a jerry_create_abort_from_value. Abort is the same as errors, except they are not caught by JS, and can only created by API or the debugger. jerry_is_error_value tells if the value is an error. Abort is considered a special variant of error. So everything true to an error is also true to an abort, but there is a special create/is_abort function to tell the difference.
This is the current plan now. We are open to feedback.

Closing beacause this planned rework had finished a long time ago.


show error instead of crashing with outdated saves

as the title says:
if there is an old safe file (or laoding doesnt work for other reasons) displaying an error message would be mroe helpful
show error instead of crashing with outdated saves

Should be handled on the new relase I'm working on.


[WIP] Implement RFC 8449 Support [1.1.1]

This adds support for the "record_size_limit" extension
defined in RFC 8449.
That is an extension to Transport Layer Security (TLS)
that allows endpoints to negotiate the maximum size of
protected records that each will send the other.
This replaces the "max_fragment_length" extension defined
in RFC 6066.
Checklist

[x] documentation is added or updated
[x] tests are added or updated

Given that this is a "feature", is 1.1.1 even appropriate? Support for 1.1.1 ends in 17 months.

I am aware that this PR won't be merged to 1.1.1, but it will nevertheless be a nice feature
in my openssl-111-features branch #11900 :smiley:
Please consider this just for reference I will come up with a PR targeting master later,
in master the state transitions will be a bit more complicated due to the
interaction with KTLS, but I would appreciate early feedback from this preview.

@t-j-h: we need a question to answer for the OTC hold.

OTC: @bernd-edlinger would you please move this PR to your personal fork on Github so we can close it?
As there is no point having it in the openssl repo when it will not ever be merged.

@openssl : outside the "record_size_limit" extension support,
parts of this PR contains some fixes, improvements and documentation that deserved to be merged to 1.1.1 .
Do you want that I highlight them ?

Note there are already two spin-off PR with some of these changes: #18252 and #18111


Cleaner way to access components and entities from entities and components

Introduce .entity and .component syntax for access if those elements.
Currently to access the first(and oftentimes only) entity that a component is attached to you need to do my_component.entities[0] and if you need to get some other component from that entity you often do my_component.entities[0].components[FF::Cmp::Other][0]. With the new proposed optional syntax this would just look like: my_component.entity.component[FF::Cmp::Other] and would function the same.
Perhaps throw a warning when you are using this syntax to access an array larger then size 1 as oftentimes programmers would want to iterate over those.

Implemented, and in a kinda cool way :^)


Custom StyleSheet ordering?

Hi, I have my tastes when ordering my StyleSheet styles props. Alphabetical order isn't good for me.
For example, I have +- at this order: flex stuff, width/height, margins, paddings, colors...
I am now doing it manually, but of course an automated way would be very nice.
What I suggest is having an array of the keys in the desired order, like ['flex', 'flexDirection', 'height', 'width']... This would be user-configurable, but I would be happy to set a default for it, if using the option to use it.
Any style in the StyleSheet not defined in that array wouldn't be sorted, or could be pushed down for an easier code.

Something like https://github.com/hudochenkov/postcss-sorting


Do not hardcode source layer

Closes https://github.com/mapbox/simplespec-to-gl-style/issues/5
This makes it so that each source name is a unique name so we do not steal/overwrite the geojson name space.
/cc @jfirebaugh

:+1:


Victory container

adds the victory-container component to wrap chart components and help streamline some props usage, especially for accessibility props
TODO:

[x] write usual prop descriptions
[x] write test

@boygirl I think this is good to go if you don't see anything missing!

@ebrillhart nicely done. Approved!


Sending Mentions is allowed by the UI but does not function on Sonoma+

For some reason, when @ someone in a gc it's doesn't get bolded like how it used to be. It appears just like typing out the name

Currently not supported on Sonoma+

Actually, it is supported. We figured out that it wasn't that it's not supported on Sonoma+, but the ddScan flag breaks on Sonoma+. So we've disabled it for those versions


Get Started with Oracle Data Safe (#700)

Updates for Cloud World

I have made the fixes requested.

Is there any way you can fix the paths in the manifest without me doing a pull request?

Is there any way you can fix the paths in the manifest without me doing a pull request?

Hi Jody! You can stash the unfinished changes, so that they are not included in your PR. Or I can submit a PR to fix the issue with manifest.json - please let me know when your commits are ready to be pushed. Thank you!

Hi @arabellayao , would you please submit a PR to fixt the issue with the manifest.json? Thanks! In the meantime, I'm going to try and get the changes completed today/tomorrow.


Button.Clicked is Action and not an event?

Why is Button.Clicked (and I assume other events) implemented as Action but not as an event?
I'd say event is more natural and standard.

Same is true of RadioGroup.SelectionChanged but not TextField.Changed

Should I create a PR to fix this by converting Action to EventHandler?

I have just never appreciated EventHandler types.   They were invented in an era that lacked lambdas.   My problem with it is two fold: the first parameter is rarely used and when you need it lambdas are now available.   The second is that the payload to the event rather than being just a value is another type, this is just too much boilerplate in my view.
Am I missing something?

The benefits of events as I see it are:

The familiarity of the pattern to people who use the .NET Framework libraries that implement events
The more obvious/less cumbersome way to attach and detatch multiple handlers
The ability to control how you store multiple handlers (by overriding the event's add and remove handlers)
The built in pattern of communicating things back to the caller or between different handlers

I'd say the latter three are pretty niche cases, though, at least in the context of a UI library. Personally, I don't have strong feelings either way, other than a bias for consistentcy

I share Miguel's scorn for events.
That said, consistency is a good thing. Sometimes the right answer is to make things consistently awful. If so, events are the perfect way to achieve that.

I feel they are as noise.
var btn = new Button();
btn.Click += () => Console.WriteLine("Clicked");

//VS
var btn = new Button();
btn.Click += (sender,args) => Console.WriteLine("Clicked");

//Long
var btn = new Button();
btn.Click += Clicked;

void Clicked() => Console.WriteLine("Clicked");

//VS
var btn = new Button();
btn.Click += Clicked;

void Clicked(object sender, EventArgs args) => Console.WriteLine("Clicked");

There are other reasons for events instead of actions besides ones mentioned by @pmsanford.
On top of my head:
Such as converting events to Tasks or using them with Reactive stuff - I think there are utilities that handle these easily with events. Not sure about actions, but it shouldn't be hard anyway.
Also, what if I want to handle multiple buttons with a single method? I'd cast sender to button and then do whatever I'd do.

Sure, it's doable. But is it optimal? Compiler creates a hidden class just for that.

Good point folks - I am on vacation this week, with limited access to the internet, but will join this conversation next week.   There seem to be two issues, one of consistency (public delegate vs event) and whether the signature should be EventHandler or the newer Action/Func pattern.
Will ponder here next week.

Introducing events would be a lie. We do not have concurrent tasks, no UI thread in the background, no devices sending something. When it comes to the terminal, we have an endless loop, polling the keyboard and the mouse - which means essentially waiting for the user - and meanwhile doing things.
Regarding the interaction between menu and a statusbar, I'd really appreciate to use new StatusBarMessage(menuItem.HelpText) to send sth to the bar. But it doesn't feel right here.
Not sure if that somehow interferes with the reactive stuff...

FWIW, I vote for events. They're standard, a lot of libraries and things are built on top of them. And I'm not sure it's possible to remove an action. Whereas with events I can remove a handler via -=.

You can remove an action by clearing it: control.Action = null
But I hear you, you want events, I need to digest this, I think I will make the change, but it will break people

Sorry I didn't see this Issue when I created this one: https://github.com/migueldeicaza/gui.cs/issues/447
I'm clearly in the EventHandler camp. For all the reasons above, plus: Action doesn't support multiple subscribers. EventHandler is designed to be used in a pub/sub way.
I think Action is awesome when used for internal-details. But for public APIs (and Terminal.Gui is a public API) EventHandler should be the norm.

Dupe of #447


[5162] Add elasticsearch tests

Covers:

Index data (CQ)
Search (CQ)
Multisearch (CQ)
GetById (CQ)
Update (CQ)
Delete (CQ)
DeleteIndex (CQ)
Bulk (CQ)
Exists (CQ)
Ping (CQ)

@jamesnetherton Can you please review ?

LGTM. Do we need tests to cover things like certificatePath or enableSSL ? Not sure as native is not tested.

Do we need tests to cover things like certificatePath or enableSSL

We could maybe follow up with that. These initial tests are mostly a reproduction of what we had for the previous elasticsearch-rest extension.

Yes, that's ok in this state. Plus if we promote it to native one day, then we could update the test coverage at this stage YAGNI style.

@aldettinger That is great idea, but given it was only for JVM, i think we are safe there. Basically all the tests are probably overkill, but i've ported all of them for future benefits. I've created followup issue for the SSL https://github.com/apache/camel-quarkus/issues/5174.

Not sure, it's overkill. Those features are present in the camel component doc, so this is good to have them covered :+1:


multiple parameter values with data_set()

I've been having trouble using data_set to set the parameters of multiple synapse or cells at once as shown in this little example:
comp = jx.Compartment()
branch = jx.Branch(comp, nseg=1)
cell = jx.Cell([branch], parents=[-1])
network = jx.Network([cell, cell, cell])

jx.fully_connect(network.cell("all"), network.cell("all"), IonotropicSynapse())

n_edges = len(network.edges)
params = [{"IonotropicSynapse_gS": jnp.ones(n_edges) * 0.001}]

# n_cells = len(network.nodes)
# params = [{"radius": jnp.ones(n_cells)}]

network.cell(0).record()

def simulate(params):

    # Set the parameters with data_set
    pstates = None
    pstates = network.IonotropicSynapse.data_set("IonotropicSynapse_gS", list(params[0].values())[0], pstates)
    #pstates = network.data_set("radius", list(params[0].values())[0], pstates)

    v = jx.integrate(network, t_max=10, param_state=pstates)
    return v

s = simulate(params)

The issue ends up being that the shape of the indices array is incompatible with the shape of the values array (they are transposes). In comments in Jaxley (base.py lines 573-577), I see that inds is supposed to be shape (num_params, num_comps_per_param), but then this becomes (1, num_synapses) or (1, num_cells) in the example above, which is incompatible with the values array which gets projected to (num_params, 1), where num_params here is the number of edges or cells.
I have tried to adjust this on the user end, and maybe there's a way to do that that I haven't tried yet, but I think it would be nice to make the code above work. I don't have the time in the next two weeks, maybe later, but of course if someone else wants to have a look please do:)

I will do some more experimenting, but I think just changing the following two lines would allow both specifying single or multiple parameter values (one for each compartment, which conveniently works out to each cell when each cell is one compartment). In _data_set() line 396:
"val": jnp.atleast_1d(jnp.asarray(val)), --> "val": jnp.atleast_2d(jnp.asarray(val)),,
same as is done with the indices, and then in get_all_parameters(), called by integrate(), line 577:
params[key] = params[key].at[inds].set(set_param[:, None]) --> params[key] = params[key].at[inds].set(set_param).
I think it is then intuitive if the values and the indices can have the same shape. I'll look into it a bit more though.

Can now be done using the select method to define indices from #447


Update dependencies

Hi @snkeegan,
While working on https://github.com/conda-forge/staged-recipes/pull/22970, we noticed that some of the dependencies do not match,
https://github.com/search?q=repo%3Aliamholtlab%2Fgemspa-spt import&type=code
It seems you packages depends on scipy and not on scikit-image
Could you fix this for the next release?
Cheers 🚀 !

Thanks!  Yes, I will fix it


added keybase and obscuredfiles

added *.keybase.io to fncuwbiisyh6ak3i.onion. You can verify this at the bottom of https://keybase.io/

Thanks for the additions!


Add cosine_simirarity_v1 which is more faster than the original cosine_similarity method.

add cosine_simirarity_v1 method which is 5 times faster than the orginal cosine_similarity method.
add cosine_simirarity_v1 method which is 5 times faster than the orgianl cosine_similarity method.
@vowelparrot
update math_utils.py file add cosine_simirarity_v1 which is more faster than the original cosine_similarity method.
the time cost test like this:
'''
x = [[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4]] * 2000
y = [[1, 2, 3, 5, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4], [1, 2, 2, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4]] * 30000
resp = cosine_similarity(x, y)
resp1 = cosine_simirarity_v1(x, y)
print('resp： ', resp)
print('score_list： ', resp1, type(resp1))
output:
Function cosine_similarity Took 1.8206 seconds
Function cosine_simirarity_v1 Took 0.3706 seconds
resp：  [[0.99663517 0.99597014 0.99663517 ... 0.99597014 0.99663517 0.99597014]
[0.99663517 0.99597014 0.99663517 ... 0.99597014 0.99663517 0.99597014]
[0.99663517 0.99597014 0.99663517 ... 0.99597014 0.99663517 0.99597014] ...
'''

@vowelparrot

Thanks for the PR! The reason we didn't use sklearn is we don't want to add it as a required dependency. We could add an import check at the top and use sklearn if it's available though.

Thanks for the PR! The reason we didn't use sklearn is we don't want to add it as a required dependency. We could add an import check at the top and use sklearn if it's available though.

if folks wanted to use this could they just import sklearn directly wherever it's actually being used? not sure how much value the wrapper adds

@hwaking Hi , could you, please, resolve the merging issues and address the last comments (if needed)? After that, ping me and I push this PR for the review. Thanks!
Please, let me know if you need help!
The example of the "on-demand import" is here.

Closing because the PR wouldn't line up with the current directory structure of the library (would need to be in /libs/langchain/langchain instead of /langchain). Feel free to reopen against the current head if it's still relevant!