Getting Started

I downloaded this project but I was not able to get it working immediately. I eventually got it working and thought I'd share how I did to save other people time and effort. Perhaps these suggestions can be added to the README or code.
To get the code to compile, I had to modify the Makefile and append chrono_time_point.cpp to the SOAP_SRC variable:
SOAP_SRC = ... $(GSOAP_CUSTOM_DIR)/chrono_time_point.cpp

To be able to test with Lingodigit ONVIF Device Tool I had to specify my computers IP (e.g. <IP_ADDRESS>) as well as the port number used by onvif_srvd (i.e. 1000) in the Add dialog:
URI: http://<IP_ADDRESS>:1000/onvif/device_service

When starting Lingodigit ONVIF Device Tool it pops up a dialog:
Please check wsdd is in /usr/local/bin/wsdd

I put the wsdd executable in /usr/local/bin/ and I also tried /usr/local/bin/wsdd/. I still get the error. I don't know why it doesn't work or how to fix it. At first I thought this might be why I couldn't connect to onvif_srvd but it doesn't seem to be needed. Once I added the port number, it worked.

Oops. I put this in the wrong project. I meant it to be added to KoynovStas/onvif_srvd.


Guidelines - Preserve precision of input data (S6)

See S.6 in guidelines.
Investigate if this is a problem, and if it is, fix.

Especially the McStas data maybe, since the counts are floating values.


Remove unused pendingTableCharactersNull dynamic property

Ticket: https://smb.myjetbrains.com/youtrack/issue/IN-429/Update-our-html5lib-php-lib-for-PHP-8.2

the only change seems to be this line:
#1 (files)

Yup. The rest is the IDE doing its job ;)

I can't tell if this property is otherwise unused.
@TheDeadJoe did you grep for it in the code base?

Yes. No occurrences. There is a property called $pendingTableCharactersDirty so at first I thought this could be just a forgotten name change, but in code it's used as a boolean value, so writing there null would make no sense.


Add cmds for global secrets

@igorkotof let's move global secrets to either of the PRs, but just one please (https://github.com/hazelops/icmk/pull/119 )

@igorkotof let's move global secrets to either of the PRs, but just one please (https://github.com/hazelops/icmk/pull/119 )

Closed due to this one  #119

Closed due to this one  #119


Suggestion: Web Editor Offline Users

Currently the Web Editor with the users argument only works on online players. Could we get a feature (maybe in the config somewhere) to also load offline users using /lp editor users?
I understand this can be nasty when you have a lot of players loading into the cache, but it'd be nice to still have this feature available if at all possible?

or the ability to load/unload an offline user?

This wouldn't efficient, considering how many players a server could have.
Imagine LP loading hundreds if not thousands of offline players for the web editor. This would either break the editor, bytebin (the site Luck made to temporary store info for/from the editor) or LuckPerms itself...
You can load a single user through /lp user <user> editor even if he's offline... For editing multiple players at once to give, remove or change permissions to a certain one use the bulk edit option.

It's a possibility with the new version - but it would be capped in terms of users.

it would be capped in terms of users.

What kind of cap are we thinking?

@Turbotailz I was thinking 1000? Sound manageable?

We can give it a go 👌

Added in v5


GPII-3916 - Change the order in the JSON fields

This PR fixes the behavior of always update these JSON definitions.
It seems that Stackdriver returns the definition of the alerts with the fields in a particular order, so when the stackdriver.rb compares the two versions always returns that needs to be updated

Sample of incident that will be solved by this PR: https://app.google.stackdriver.com/incidents/0.laavenut0uow?project=gpii-gcp-stg

LGTM!


feat: avm/res/app/managed-environment storage

Description
New feature: possibility to add storages on a container apps environment.
Pipeline Reference
Unfortunately I cannot make the NFS storage mount work. Should I remove this option for the time being?
Otherwise, everything works.
I tried to replicate the configuration from https://learn.microsoft.com/en-us/azure/container-apps/storage-mounts?tabs=nfs&pivots=azure-resource-manager#configuration-1 but I can't find the correct syntax for the fileShareName (which from the errors I am facing should be 'properties.nfsAzureFile.shareName' must be in format '/<accountName>/<fileShareName>' and the fileShareName must be from 3 through 63 characters long)

Pipeline

Type of Change

[ ] Update to CI Environment or utilities (Non-module affecting changes)
[ ] Azure Verified Module updates:

[ ] Bugfix containing backwards-compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in version.json:

[ ] Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description.
[ ] The bug was found by the module author, and no one has opened an issue to report it yet.

[X] Feature update backwards compatible feature updates, and I have bumped the MINOR version in version.json.
[ ] Breaking changes and I have bumped the MAJOR version in version.json.
[ ] Update to documentation

Checklist

[X] I'm sure there are no other open Pull Requests for the same update/change
[x] I have run Set-AVMModule locally to generate the supporting module files.
[ ] My corresponding pipelines / checks run clean and green without any errors or warnings

@hundredacres @arnoldna would you have some time to please review this PR?

@arnaud-tincelin do you have a pipeline badge you can add so we can confirm the changes are working in your branch?

@hundredacres @arnoldna would you have some time to please review this PR?

Yes

@arnoldna @hundredacres any news about merging this?

@arnaud-tincelin Sorry for the delay in getting back to you. If you could remove the unused output from the dependancies file, we can merge this change in.

@hundredacres done


Fix for Memory Leak in cdist and Metric-Like Functions

Fix for Memory Leak in cdist and Metric-Like Functions
alert(gpt)
Problem Description
I identified a memory leak issue in the cdist function, as well as other metric-like functions, within our codebase. This leak becomes evident in scenarios where these functions are called repeatedly, such as within a while True loop. An example to demonstrate the issue is as follows:
while True:
    simsimd.cdist(embeddings, query)

This code snippet leads to a gradual increase in memory usage over time, indicating a memory leak.
Root Cause
The memory leak stems from the improper handling of numpy arrays, particularly in the context of garbage collection (GC). The issue arises when PyArray_NewFromDescr is used with non-null data. In such cases, the data must remain alive for the entire lifetime of the array. However, it is crucial to allow both numpy and Python's garbage collector to deallocate the array when it is no longer in use.
Solution
According to the numpy C-API documentation (Numpy C-API Array From Scratch), the constant NPY_ARRAY_OWNDATA should not be used manually. The documentation states:
"The data area is owned by this array. Should never be set manually, instead create a PyObject wrapping the data and set the array’s base to that object. For an example, see the test in test_mem_policy."
Following the guidelines from the numpy documentation, I referred to a specific implementation in the numpy repository for guidance. The reference code can be found here: Numpy Test Memory Policy. By aligning our implementation with the approach demonstrated in this example, I was able to resolve the memory leak issue.

I may have a similar issue in other repos, like USearch, so finding a way to avoid the extra allocation would be great, @smthngslv 🤗

@smthngslv how did you catch the leak? which tool would you suggest for the Python binding? Maybe something we can add in CI?
PS: I'm gonna merge the current solution and looking forward to the one that adds the out parameter, in case you are still onto it 🤗

:tada: This PR is included in version 3.6.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

@smthngslv how did you catch the leak? which tool would you suggest for the Python binding? Maybe something we can add in CI?

Just manual testing, comment out some lines and testing again. About the build process - I use scikit-build and pybind11 with cmake (adopted from https://github.com/pybind/scikit_build_example). Kinda love pybind11, since you do not need to deal with low-level python c api:
# Find the module development requirements (requires FindPython from 3.17 or scikit-build-core's built-in backport).
find_package(Python REQUIRED COMPONENTS Interpreter Development.Module)
find_package(pybind11 CONFIG REQUIRED)
find_package(OpenMP)

# Add a library using FindPython's tooling (pybind11 also provides a helper like this).
python_add_library(_core MODULE src/main.cpp WITH_SOABI)
target_link_libraries(_core PRIVATE pybind11::headers)
if(OpenMP_CXX_FOUND)
    target_link_libraries(_core PUBLIC OpenMP::OpenMP_CXX)
endif()

looking forward to the one that adds the out parameter, in case you are still onto it 🤗

Yep, I am still on it, as soon as I manage to find some free time :)


feat: 5586 - OxF filter for term searches

What

Now for term searches, there is an OxF "radio button": the search will be run on the specific server.
Pending questions

for the user page ("products you edited") and contribute button ("contribute by completing products"), for the moment we consider only the food products, does that make sense? (impact on user_preferences_account.dart, user_preferences_contribute.dart)
Maybe reinit default product type (food) with latest value for the filter (impact on search_product_helper.dart)
Matomo: shouldn't we send the product type to Matomo? (impact at least on product_query.dart)
I believe it's very optimistic to display 4 labels in the filter - what about icons and hints? (impact on product_query.dart)

Screenshots

food
beauty

pet food
products

Part of

#5586

Impacted files

category_product_query.dart: refactored with new productType parameter
keywords_product_query.dart: refactored with new productType parameter
paged_product_query.dart: refactored with new productType parameter
paged_search_product_query.dart: refactored with new productType parameter
paged_to_be_completed_product_query.dart: refactored with new productType parameter
paged_user_product_query.dart: refactored with new productType parameter
product_list.dart: refactored with new productType parameter
product_query.dart: added a "get label" method for ProductType
search_field.dart: displayed the new optional "additional filter" (e.g. OxF)
search_helper.dart: added a new optional "additional filter"
search_product_helper.dart: new "product type filter" widget
summary_card.dart: minor "productType" fix
user_preferences_account.dart: added "productType.food" to user page counts
user_preferences_contribute.dart: added "productType.food" to "to be completed" products

Codecov Report
Attention: Patch coverage is 0% with 66 lines in your changes missing coverage. Please review.

Project coverage is 6.88%. Comparing base (4d9c7fc) to head (0cdaa73).
Report is 334 commits behind head on develop.

Files with missing lines
Patch %
Lines

...th_app/lib/pages/search/search_product_helper.dart
0.00%
18 Missing :warning:

...ages/smooth_app/lib/pages/search/search_field.dart
0.00%
12 Missing :warning:

...s/smooth_app/lib/query/category_product_query.dart
0.00%
7 Missing :warning:

...s/smooth_app/lib/query/keywords_product_query.dart
0.00%
7 Missing :warning:

...lib/query/paged_to_be_completed_product_query.dart
0.00%
6 Missing :warning:

...smooth_app/lib/query/paged_user_product_query.dart
0.00%
5 Missing :warning:

...kages/smooth_app/lib/data_models/product_list.dart
0.00%
2 Missing :warning:

...ges/smooth_app/lib/pages/product/summary_card.dart
0.00%
2 Missing :warning:

...ages/smooth_app/lib/query/paged_product_query.dart
0.00%
2 Missing :warning:

...ib/pages/preferences/user_preferences_account.dart
0.00%
1 Missing :warning:

... and 4 more

Additional details and impacted files
@@            Coverage Diff             @@
##           develop   #5637      +/-   ##
==========================================
- Coverage     9.54%   6.88%   -2.67%
==========================================
  Files          325     402      +77
  Lines        16411   21307    +4896
==========================================
- Hits          1567    1466     -101
- Misses       14844   19841    +4997

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Agree with the first 3 points, on 4th, I'd like to try  text first. You could have chips instead of a big pill so that we can overflow on very small screens.

Thank you @teolemon for your review!
I'm about to transform somehow the pending questions into issues.
Please keep in mind that we still cannot edit non-food products - that would probably create food products from scratch.


Problem with serializing custom date fields that are empty strings

Laravel Version: 5.5.45 (but this also seems to be present in 5.8)
PHP Version: 7.2.13
Database Driver & Version: mysqlnd 5.0.12-dev - 20150407

Description:
A pretext to this was fixed in #22107, which was reported in #21338.
Carbon will still throw an exception when trying to convert a collection via a HasMany relation of a model where a date field is an empty string (and not NULL). The date field (order_date) is of the type varchar in the DB due to a custom format (see #22385 for that).
Example
Project::find(1)->orders->toArray()

Returns: InvalidArgumentException with message 'Data missing'.
It fails when one of the order's order_date is set to an empty string instead of NULL.
After looking through the stack trace I found that it is likely due to addDateAttributesToArray() in Illuminate/Database/Eloquent/Concerns/HasAttributes.php, where the isset() check would catch NULLed values, but not "" (empty strings).
Steps To Reproduce:
Create project and order tables
Schema::create('project', function(Blueprint $table) {
    $table->increments('id');
    $table->string('name');
    $table->timestamps();
});

Schema::create('order', function(Blueprint $table) {
    $table->increments('id');
    $table->string('name');
    $table->string('order_date');
    $table->unsignedInteger('project_id');
    $table->timestamps();
});

(Note the order_date which is string).
Create models
class Project extends Model
{
    protected $table = 'project';

    public function orders() {
        return $this->hasMany(Order::class);
    }
}

class Order extends Model
{
    protected $table = 'order';
    protected $dates = ['order_date'];
    protected $dateFormat = 'YmdHis';
}

Create seeders
public function run()
{
  Project::create([
    'name' => 'Test project 1',
  ]);
  Order::create([
    'name' => 'Test order',
    'order_date' => now(),
'project_id' => 1,
  ]);
  Order::create([
    'name' => 'Test order',
    'order_date' => '',
    'project_id' => 1,
  ]);
}

Run tinker
Tinker is able to dump the collection like this:
>>> App\Project::find(1)->orders
=> Illuminate\Database\Eloquent\Collection {#2876
     all: [
       App\Order {#2877
         id: "1",
         name: "Test order",
         order_date: "20190328124507",
         project_id: "1",
         created_at: "20190328124507",
         updated_at: "20190328124507",
       },
       App\Order {#2884
         id: "2",
         name: "Test order",
         order_date: "",
         project_id: "1",
         created_at: "20190328124507",
         updated_at: "20190328124507",
       },
     ],
   }

But if I try to convert it to an array, it fails:
>>> App\Project::find(1)->orders->toArray()
InvalidArgumentException with message 'Data missing'

Workaround
Overriding addDateAttributesToArray() in Order model, so that the model looks like this:
(...)
protected $table = 'order';
protected $dates = ['order_date'];
protected $dateFormat = 'YmdHis';

protected function addDateAttributesToArray(array $attributes)
{
    foreach ($this->getDates() as $key) {
        if (! isset($attributes[$key])) {
            continue;
        }
        // Added this if check. Could possibly be more speficic, i.e.:
        // if ($attributes[$key] === '')
        if (empty($attributes[$key])) {
            continue;
        }

        $attributes[$key] = $this->serializeDate(
            $this->asDateTime($attributes[$key])
        );
    }

    return $attributes;
}

Thank you for the detailed issue 👍
If you're up for it, feel free to create a PR with a test for this. Otherwise I'll pick this up once I get around to it.

If you send this in to the 5.5 branch you can also send this separately to the 5.8 branch.

I would love to submit a PR, but I have very little exercise in writing tests, especially for Laravel as a framework. I don't currently see any unit tests covering the addDateAttributesToArray method, but I also wouldn't know where to put it if it should be added :slightly_frowning_face:
Also: edited my description. I was testing this on Laravel 5.5.45, not 5.5.42 as initially reported.

Seems like a PR was attempted. @khatriafez please note that every PR needs a detailed explanation and not just a link to an issue. This is noted in the PR template: https://raw.githubusercontent.com/laravel/framework/5.8/.github/PULL_REQUEST_TEMPLATE.md

I'm closing this because it has been determined that an empty string simply isn't a valid date. The thrown exception is wanted. Please see https://github.com/laravel/framework/pull/30549#issuecomment-552085930 for more info.


WalletConnect bottom sheet

Description
This PR is mainly UI + refactoring. No real functional changes are made.

make the WalletConnect screens look like a bottom sheet
styles update for the bottom sheet: the image now shows the valora icon too, content is left aligned, transaction data will be shown in key/value pairs
ensure that WalletConnect connection + action requests are sharing the same presentational component

To come in follow up PRs:

exposing transaction data that wasn't exposed before
native bottom sheet
reference current dapp list
make dappkit dapps also share this bottom sheet

Other changes
N/A
Tested
Updated unit tests + e2e.
How others should test
Connect to dapps that are using WalletConnect (MentoFi, ChurritoFi) - ensure that connection and making transactions work as before.
Related issues

Relates #2342

Backwards compatibility
Yes

Great work! 👍
In follow up PRs, can we also center the the activity indicator and use snap points for the bottom sheet so there's less white content by default?
Also would be great to have a way to nicely display all types of dapp icons so they don't look too weird. The current solution with the circle isn't great for icons which are squares or don't have enough padding already around them. I know this isn't specific to the work here, just mentioning it so we keep it in the back of our minds.

yeah definitely, good callouts...
i updated the bottom sheet to vertically center the loading spinner now. for the snap point, i figure it will come for free in react-navigation-bottom-sheet so did not spend time on it now, but it is coming :D
and i will check with Nitya about the circles - we did have this conversation for the recently used dapps too, and only went with circles as an MVP. i think it's a good time to think about this again

Hey @MuckT we have verified the above task on Latest iOS TestFlight build V:1.35.0 and updated status in Test Rails
Observation : User is able to connect to wallet and able to perform transaction however there are different screens appearing and it is creating confusion hence we have raised a query for it
New Query : link
Test rail Links:
iOS 14 : https://valoraapp.testrail.io/index.php?/tests/view/47340
Device : iPhone 12(14.7.1)
Cc : @kathaypacific
Thanks...!

Hey @MuckT, we have verified the above task on Android Internal Release Build V:1.35.0 and updated status in Test Rails
Test rail Links:
Android 10 : https://valoraapp.testrail.io/index.php?/tests/view/47402
Devices:  Samsung Galaxy M41 (10.0)
CC : @kathaypacific
Thanks...!!


"Add Exclusions" section could be improved

Hi, I was expecting the doc to show me how to add exclusions, it seems to only show how to tell what would happen if the current detection was added as an exclusion.
In reality, you have to go to MEM admin center, Endpoint security, select your ASR Rules, Edit configuration, scroll to the bottom and add the path(s) there.
Perhaps the article could point us in the right direction in future? cheers

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 498a23d9-d5b0-2c98-edfa-d9871675f5ae
Version Independent ID: 498a23d9-d5b0-2c98-edfa-d9871675f5ae
Content: Test attack surface reduction (ASR) rules
Content Source: microsoft-365/security/defender-endpoint/attack-surface-reduction-rules-deployment-test.md
Product: m365-security
Technology: mde
GitHub Login: @jweston-1
Microsoft Alias: v-jweston

@msbemba

@SpitFire-666 Thank you for your feedback. The exclusion is a part of the implement ASR rules document.
You can exclude files and folders, by group policy,  via PowerShell, and also using MDM CSPs to exclude files and folders.
https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/attack-surface-reduction-rules-deployment-implement?view=o365-worldwide#customize-attack-surface-reduction-rules
Hope this helps!
Thanks
Sri

@msbemba Thank you very much for the contribution and sharing this explanation. @SpitFire-666 Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Clean warnings.

Is your feature request related to a problem? Please describe.
Currently, there are over 200 warning in the solution.
Describe the solution you'd like
Let's clean all those warnings.

Done https://github.com/reactiveui/ReactiveUI.Validation/pull/13


Obstructed start-point

I've been using your library for pathfinding in a 2D simulation for quite some time. I'm working on implementing the v2 version but encountered a silly issue where no path is found due to obstructed start position.
This makes sense because consider that a unit starts off at position 5,5. When the algorithm tries to find a path, the position 5,5 would be occupied and instantly return NO_PATH found
This could potentially be remedied by a skip paramter?
By commenting out the following lines, the code works as expected (or atleast the start position part of the if statement)
Line 1303-1305
            //if(!grid(start.x, start.y) || !grid(end.x, end.y))
            //    return JPS_NO_PATH;

The old version had exactly this check as well. If it worked there this must have been by pure chance.
I mean sure you can comment that out to make that work but to me this looks like a problem in your code that needs to be fixed. Starting pathfinding from inside a wall and expecting it to work doesn't make sense to me.
Maybe i'm misunderstanding what you're doing there? A unit(?) blocking itself? Why is a unit part of your map data?

I'll try to make my case a bit more clear. Sorry!
Below is a graphical representation of the game I'm implementing your JPS Implementation.

As you stated, the units are part of the map-data (grid). The reason for this is that a tile/cell is not "walkable" if occupied by another unit. When the algorithm starts, it checks if the start-tile is obstructed, and this is always true because the unit already stands on the start position.
Also in order for the algorithm to have any practical use in such 2-dimensional games, it has to evaluate if the path is obstructed by other units.

Say I would evaluate the blue unit to move towards the upper right corner of the map. Here, the blue unit would simply move through the enemy if "other units" were not part of the map data.
Other units are simply treated as walls, and is, therefore, a non-issue, But because there is no way identify which unit the "findPath" request came from I cannot make exceptions for the first step of tile path calculation.
I'm not sure why this worked without issue in the first version. I did, however, not encounter this issue before doing an in-place replacement of the header file.
It is a non-issue for me to remove the first part of the if-statement mentioned earlier, but I feel it natural to have an option to skip obstruction of the start position.

Hi,
I've tested the new version. Without the flags, it does not move at all (I suppose this is expected for my case)
There is some weird behavior when calculating the path is calculated:
s denotes  start
g denotes goal
p denotes JPS path
Standing at 1,1 clicking at 13,1
sx=1, sy=1, gx=13, gy=1
px=1, py=24
px=1, py=23
px=1, py=22
px=1, py=21
px=1, py=20
px=1, py=19
px=1, py=18
px=1, py=17
px=1, py=16
px=1, py=15
px=1, py=14
px=1, py=13

Standing at 1,1 clicking at 1,13 (lower left)
sx=1, sy=1, gx=1, gy=13
px=24, py=1
px=23, py=1
px=22, py=1
px=21, py=1
px=20, py=1
px=19, py=1
px=18, py=1
px=17, py=1
px=16, py=1
px=15, py=1
px=14, py=1
px=13, py=1

I've double checked that the grid collision is represented correctly (Worked before with JPS2)

        JPS::findPath(path, unit.getPlayer().getGame().tilemap, unit.tile->x, unit.tile->y, goal->y, goal->x, 1, (JPS_Flag_NoStartCheck | JPS_Flag_NoEndCheck));
        std::cout << "sx=" << unit.tile->x << ", sy=" << unit.tile->y << ", gx=" << goal->x << ", gy=" << goal->y << std::endl;
        for(auto pathItem : path) {
            std::cout << "px=" << pathItem.x << ", py=" << pathItem.y << std::endl;
        }

        // Insert found path to the walking path vector
        std::reverse(path.begin(), path.end());
        for(auto pathItem : path) {
            Tile& tile = unit.getPlayer().getGame().tilemap.getTile(pathItem.x, pathItem.y);
            unit.walking_path.push_back(&tile);
        }

And thanks for the support by the way! Much appreciated!

        JPS::findPath(path, unit.getPlayer().getGame().tilemap, unit.tile->x, unit.tile->y, goal->y, goal->x, 1, (JPS_Flag_NoStartCheck | JPS_Flag_NoEndCheck));

(..., goal->y, goal->x, ...)

Did you by chance switch x and y?
And why do you reverse the path? It should deliver the path points in the right order already.
I just did a quick check by replicating your level as ascii-art and it appears to work fine here.
But you're not the only one with problems with the new version, so there must be something wrong that i'm just too dumb to see. It should definitely not go out-of-bounds like this (assuming your grid check function is correct). I assume your map is non-wraparound?

Hi,
The x and y should be correct and i cannot recollect how i defined the grid check (ill check later today).
Nontheless, the exact same codebase runs fine on the older version.
Ill come back to you later today with additional details and see if i can identify any reason behind the behavior

There was a bug, fix is up! Pull + test again.

@fgenesis works perfectly!

I'll close this help ticket and query you if I find any funnies.
You have done an incredible job with this library and I thank you for your continued maintaining!


Avoid erroneous XLS format detection for XLSX files

Fixes #6361
In previous versions of the code, the order of the arrays in which we were storing the supported extensions was not taken into account.
This was causing that the format XLS was being considered for some XLSX files, as it was the first match of the URL_FILENAME_REGEX.
Not the supported extensions array gets ordered by length of the extensions name so that the regex is greedy:
CartoDB::Importer2::Unp::SUPPORTED_FORMATS
                                  .concat(CartoDB::Importer2::Unp::COMPRESSED_EXTENSIONS)
                                  .sort_by { |s| s.length }.reverse

This PR also contains code changes to move the regex related constants to class methods.
Finally, a new test has been added to ensure that the XSLX extension is being detected correctly in the scenario where it wasn't previously. The old version of the code makes this test to fail.

Please CR @rafatower

Just had a minor comment, other than that :+1:

Will deploy tomorrow morning.


New Revel project doesn't run or build

Revel modules depends on github.com/myesui/uuid, which doesn't exist. Any attempts to run or build a newly-created revel project fails:
[caeril@localhost src]$ revel new foobar
Revel executing: create a skeleton Revel application
Your application has been created in:
   /home/caeril/work/go/src/foobar

You can run it with:
   revel run -a  foobar
[caeril@localhost src]$ revel run -a foobar
Revel executing: run a Revel application
Downloading related packages ...ERROR 10:30:07 command_config.go:250: Failed to import package                 error="exit status 1" gopath=/home/caeril/work/go GO-ROOT=/usr/lib/golang output="go: finding module for package github.com/revel/modules\ngo: finding module for package github.com/stretchr/testify/assert\ngo: finding module for package gopkg.in/yaml.v2\ngo: finding module for package github.com/BurntSushi/toml\ngo: finding module for package github.com/myesui/uuid\ngo: finding module for package gopkg.in/stretchr/testify.v1/assert\ngo: finding module for package github.com/myesui/uuid/savers\ngo: found github.com/revel/modules in github.com/revel/modules v1.0.0\ngo: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.7.0\ngo: found github.com/BurntSushi/toml in github.com/BurntSushi/toml v0.3.1\ngo: found gopkg.in/yaml.v2 in gopkg.in/yaml.v2 v2.4.0\ngo<EMAIL_ADDRESS><EMAIL_ADDRESS>reading github.com/myesui/uuid/go.mod at revision v1.0.0: unknown revision v1.0.0\n"
 completed.
Downloading related packages ...ERROR 10:30:09 command_config.go:250: Failed to import package                  error="exit status 1" gopath=/home/caeril/work/go GO-ROOT=/usr/lib/golang output="go: finding module for package github.com/revel/modules\ngo: finding module for package github.com/stretchr/testify/assert\ngo: finding module for package github.com/myesui/uuid/savers\ngo: finding module for package gopkg.in/stretchr/testify.v1/assert\ngo: finding module for package github.com/myesui/uuid\ngo: finding module for package github.com/BurntSushi/toml\ngo: finding module for package gopkg.in/yaml.v2\ngo: found github.com/revel/modules in github.com/revel/modules v1.0.0\ngo: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.7.0\ngo: found github.com/BurntSushi/toml in github.com/BurntSushi/toml v0.3.1\ngo: found gopkg.in/yaml.v2 in gopkg.in/yaml.v2 v2.4.0\ngo<EMAIL_ADDRESS><EMAIL_ADDRESS>reading github.com/myesui/uuid/go.mod at revision v1.0.0: unknown revision v1.0.0\n"
 completed.
ERROR 10:30:10   file.go:341: Unable to import this package            package=foobar/app
ERROR 10:30:10   file.go:341: Unable to import this package            package=github.com/revel/revel
ERROR 10:30:10  revel.go:107: Unable to execute                        error="Revel paths[error No revel found]"

Right now I'm getting around this by copying and modifying my existing revel projects, but new ones neither run nor build.

I'm seeing the same behavior while trying to make an build machine.  I am almost certain it has to do with the twinj depending on myesui/uuid.  Looking around myesui has completely disappeared and twinj is archived and read only.  I'm going to copy the entire contents for the directories and hope it will build when it gets to the other machine.

Has anyone found a solution to this yet? Are there any plans to migrate to something other than twinj?

Ok, I got around this issue by just forking, renaming, and replacing twinj/myesui with google's uuid package.
The fork/rename was required only because I couldn't get the ridiculous go 11 modules system to stop re-downloading during a go clean -modcache after replacing the uuid dependency - it continued to demand the existence of the broken dependency unless the module name was changed.

@caeril Did you push a branch with this change?  I'm trying to setup docker for my project at the moment.  I forgot about this issue until I tried to pull everything from scratch.

@caeril Did you push a branch with this change? I'm trying to setup docker for my project at the moment. I forgot about this issue until I tried to pull everything from scratch.

No, but I have my own fork with the workaround here: https://github.com/caeril/frevel
But please don't rely on it. For all I know I might just disappear like the dependency we're talking about, haha.

Well, it's been quite a couple years. Sad to see Revel is having issues like this. I'll do my best to get back into maintaining this project to at least merge key PRs like this. If anyone else has time to volunteer to assist, that'd be welcome!


Create platinum-elements for polymer 0.17

Part of #533

Ended up creating just one repo/package for all of these, pushed to github https://github.com/dart-lang/polyme but not yet published. Also still needs tests/verification, but I will create separate bugs for that.


compression is applied on sql string instead of file

When I create a db dump using the --compress flag or enabling the compress to true from the config file the output file cannot be uncompressed successfully. I tried different apps to do it, here are the results I got
Archive Utility App

This creates another compressed file with .cpgz as the extension. Ex: source file is db.sql.gz the extracted file from Archive Utility App will be db.sql.gz.cpgz. If I try to unzip this one it will create another file called db.sql 2.gz which will then become db.sql 2.gz.cpgz if i uncompress that again therefore becoming an infinite loop if I keep doing it

RAR Extractor Lite

The app will crash

Keka

Extraction failed.. will create a folder with the same name of the gz filed but is empty folder

Tools from CLI
dtrx

dtrx: ERROR: could not handle ets_20180828105530.sql.gz
dtrx: ERROR: treating as gzip-encoded compressed file failed: decoding error: 'zcat' returned status code 1
dtrx: ERROR: Error output from this process:
zcat: unknown compression format

tar
-tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Im running on MAC OS high sierra

this is because I have compressed the result string before saving in the file that's why you get an error when you try to decompress it. Code need some love and compress the sql file and not the sql string that will be saved on it. I'll try to find some time this weekend to work on that or if you have managed to fix it feel free to send a PR.

@isaac-manubag this is fixed on new version 4.* please check new command fix-file for more datailed usage.


How do I deploy in other languages?

On this page it says "Download the Office Deployment Tool from the Microsoft Download Center."
On that page (https://www.microsoft.com/en-us/download/details.aspx?id=49117) it says "Important! Selecting a language below will dynamically change the complete page content to that language." and below that; Language: English.
I want to deploy in Danish but I can't find a way to change the language from English on, where I download the Office Deployment Tool.
Help!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 66127c02-c2dc-f6d1-6e83-a6d2b1166731
Version Independent ID: 27635315-22db-d136-f07e-5169f6207b50
Content: Overview of the Office Deployment Tool
Content Source: DeployOffice/overview-of-the-office-2016-deployment-tool.md
Service: o365-proplus-itpro
GitHub Login: @jwhit-MSFT
Microsoft Alias: jwhit

@RipRapRob. you can download the proofing tools for all the languages and then install with an administrator rights
go to the below website and click the radio button
and then select the danish language
https://support.office.com/en-us/article/Language-Accessory-Pack-for-Office-82ee1236-0f9a-45ee-9c72-05b026ee809f

@RipRapRob. ODT is available only English only. But you configure the xml file and deploy office  in danish language
just by replacing the "en-us"   to "da-dk" in the following configuration files
configuration-Office365-x64.xml
configuration-Office365-x86.xml
configuration-Office2019Enterprise.xml
you can edit by using notepad, after editing, you must save as .xml

@RipRapRob -- @RAJU2529  is correct. The Office Deployment Tool itself is English only, but you can modify the sample xml file to install Office in Danish.
If you need to deploy multiple languages in an organization, see this article for more information -- Overview of deploying languages in Office 365 ProPlus.


x/tools/cmd/gorename: cgo pkg-config not supported by loader(?)

(Sorry for the terrible issue description, I don't really understand all the pieces.)
When I use gorename on a project that transitively depends on a cgo library, I get the error:
cgo pkg-config not supported

I get this in go version go1.4.2 linux/amd64 but also on my other box which has a newer version of go.
I hear that @alandonovan knows about this.  I find the error in this module:
https://code.google.com/p/go/source/browse/go/loader/cgo.go?repo=tools

For what it's worth, I'd be fine with any simple workarounds you have -- the cgo module is not important to the code I'm editing.  For example if I could check in the cgo-generated files to fix this, or use a build tag to remove the cgo files when using gorename, etc... (Obviously just fixing it is preferable though if that's easy.)
...which gives me an idea!  I might try inlining the results of pkg-config into cflags/ldflags on the relevant source.

I have the same problem.
gorename output:

cgo pkg-config not supported
cgo pkg-config not supported
cgo pkg-config not supported
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/xml/attribute.go:13:3: undeclared name: XmlNode
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/xml/cdata.go:11:3: undeclared name: XmlNode
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/xml/comment.go:4:3: undeclared name: XmlNode
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/xml/element.go:4:3: undeclared name: XmlNode
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/xml/pi.go:4:3: undeclared name: XmlNode
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/gokogiri.go:21:38: undeclared name: html
/Users/leo/src/mygo/src/github.com/moovweb/gokogiri/gokogiri.go:33:37: XmlDocument not declared by package xml
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:12:11: Node not declared by package xml
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:80:41: undeclared name: xpath
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:18:19: undeclared name: xpath
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:18:38: undeclared name: xpath
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:31:22: Node not declared by package xml
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:67:32: undeclared name: xpath
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:67:55: Node not declared by package xml
/Users/leo/src/mygo/src/github.com/DSrcl/Microdata-Parser/parser.go:129:24: Node not declared by package xml

same for me with https://github.com/abh/geoip

This is a bug in golang.org/x/tools/go/loader. The fix requires adding logic to the runCgo function so that it executes pkg-config, following the logic in the go build command (see $GOROOT/src/cmd/go/build.go).

Is there an issue for this bug, or is this the issue?  Is there a patch somewhere that I might be able to apply locally while waiting for this to make it into the release?

@alaska This is the issue.  I don't think there is an existing patch.  Alan outlined above what needs to happen.

Any luck here, its been open a while and doesn't seem to be scheduled. I am seeing while trying to use tools like errcheck or unused.

Patch: https://go-review.googlesource.com/#/c/21121/


func in uretprobe returns caller name

Originally discovered by @xutao323.
Using the func builtin inside an uretprobe gives the name of the caller function. However, the documentation says:

func - Name of the traced function

which suggests that the name of the callee (one that the uretprobe is attached to) should be returned.
The reason for this is that func reads the ip register which is at that time set to the return instruction in the caller function. This should be either fixed or at least marked in documentation.
What reproduces the bug?
# cat test.c
#include <unistd.h>

int fun() { }

int main() {
    fun();
    sleep(1);  // this is necessary due to other bug, see issue #2284
}

# bpftrace -e 'uretprobe:test:fun { printf("%s\n", func); }'
Attaching 1 probe...
main

Some discussion here: https://lore.kernel.org/bpf/ZKuyKj8jrEPzWYMM@krava/T/#u

FYI, kernel now supports get_func_ip for uprobes (https://github.com/torvalds/linux/commit/a3c485a5d8d4)

Got a chance to revisit. I tried recent Fedora kernel-6.7.0-0.rc4.35.fc40 with #2645, and this issue can be resolved.
My steps:

Install latest Fedora kernel: the get_func_ip change is in newer kernels after 6.6, I used kernel-6.7.0-0.rc4.35.fc40
Build #2645 locally: a small change to pass get_func_ip for both uprobe and kprobe on top of https://github.com/ajor/bpftrace2/commit/b8a14f44051ce4ece7e34b0062ca9be58e3ab9ea (test still fails, i don't know):

diff --git a/src/ast/passes/codegen_llvm.cpp b/src/ast/passes/codegen_llvm.cpp
index 92ea1fe7..db288ab9 100644
--- a/src/ast/passes/codegen_llvm.cpp
+++ b/src/ast/passes/codegen_llvm.cpp
@@ -278,7 +278,7 @@ void CodegenLLVM::visit(Builtin &builtin)
     // But only use the it for kretprobes, as the helper will fail for kprobes
     // placed within a function (not at the entry).
     auto probe_type = probetype(current_attach_point_->provider);
-    if (probe_type == ProbeType::kretprobe)
+    if (probe_type == ProbeType::kretprobe || probe_type == ProbeType::uretprobe)
     {
       expr_ = b_.CreateGetFuncIp(ctx_, builtin.loc);
     }
diff --git a/src/ast/passes/semantic_analyser.cpp b/src/ast/passes/semantic_analyser.cpp
index 4fdba408..e0530972 100644
--- a/src/ast/passes/semantic_analyser.cpp
+++ b/src/ast/passes/semantic_analyser.cpp
@@ -357,6 +357,7 @@ void SemanticAnalyser::visit(Builtin &builtin)
             << "The func builtin can not be used with '"
             << attach_point->provider << "' probes";

+      /*
       if (type == ProbeType::uretprobe)
       {
         // Disabled due to #2285
@@ -364,6 +365,7 @@ void SemanticAnalyser::visit(Builtin &builtin)
             << "The 'func' builtin is not available for uretprobes. Consider "
                "using 'probe' instead.";
       }
+       */

       if (type == ProbeType::kretprobe &&
           !bpftrace_.feature_->has_helper_get_func_ip())
diff --git a/tests/semantic_analyser.cpp b/tests/semantic_analyser.cpp
index 47f485ea..2b338d96 100644
--- a/tests/semantic_analyser.cpp
+++ b/tests/semantic_analyser.cpp
@@ -794,7 +794,7 @@ TEST(semantic_analyser, call_func)
   test("uprobe:/bin/sh:f { printf(\"%s\", func);  }", 0);

   test("kretprobe:f { func }", 0);
-  test("uretprobe:/bin/sh:f { func }", 1); // Disabled due to #2285
+  test("uretprobe:/bin/sh:f { func }", 0); // Disabled due to #2285

   MockBPFfeature nofeature(false);
   test(nofeature, "kretprobe:f { func }", 1);

The local built binary:

# ./bpftrace --info
System
  OS: Linux 6.7.0-0.rc4.35.fc40.aarch64 #1 SMP PREEMPT_DYNAMIC Mon Dec  4 15:48:19 UTC 2023
  Arch: aarch64

Build
  version: v0.18.0
  LLVM: 16.0.6
  unsafe probe: no
  bfd: yes
  libdw (DWARF support): yes

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: no
  get_boot_ns: yes
  dpath: yes
  skboutput: yes
  get_func_ip: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf: yes
  module btf: yes
  map batch: yes
  uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes
  ringbuf: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes
  kprobe_multi: no
  raw_tp_special: yes
  iter: yes

Test uretprobe func: local built binary is correct now, while old version system binary still points to ip register

[root@fedora src]# ./bpftrace -e 'uretprobe:/root/test/test:fun { printf("%s\n", func); }'
Attaching 1 probe...
fun
^C

[root@fedora src]# bpftrace -e 'uretprobe:/root/test/test:fun { printf("%s\n", func); }'
Attaching 1 probe...
main
^C

cheers

Related issue (just adding for reference): https://github.com/iovisor/bpftrace/issues/68

Tested kretprobe func in https://github.com/iovisor/bpftrace/issues/2285#issuecomment-1849465762 environment, #2645 is able to fix #68 as expected. So func works in both kretprobe and uretprobe now.
#2645 test:
[root@fedora src]# bpftrace -e 'kretprobe:vfs_read { printf("func: '%s'\n", func); exit(); }'
Attaching 1 probe...
func: ksys_read
Lost 3859 events

[root@fedora src]# ./bpftrace -e 'kretprobe:vfs_read { printf("func: '%s'\n", func); exit(); }'
Attaching 1 probe...
func: vfs_read
Lost 3849 events

#68 test:
[root@fedora src]# bpftrace -e 'kretprobe:vfs_* { @[func] = count(); }'
Attaching 76 probes...
^C

@[ksys_write]: 1
@[ksys_write]: 2
@[ksys_read]: 3
@[vfs_fstatat]: 22
@[__do_sys_newfstatat]: 23
@[vfs_fstat]: 72
@[vfs_getattr]: 73
@[do_open]: 144
@[ksys_read]: 179
[root@fedora src]# ./bpftrace -e 'kretprobe:vfs_* { @[func] = count(); }'
Attaching 76 probes...
^C

@[vfs_write]: 2
@[vfs_fstat]: 64
@[vfs_fstatat]: 65
@[vfs_getattr]: 72
@[vfs_getattr_nosec]: 73
@[vfs_open]: 144
@[vfs_read]: 254

@xutao323 Thanks for re-testing. We'll try to get that PR merged.


Upgrade fails with refusing downgrade

When user upgrades and we accidentally had lower version

For security reasons, updates that downgrade version of the application are not allowed. Refusing to downgrade app from version 2.0 to 1.0.3. Aborting update.

This can happen if we find a problem with a release and take it down, for example.

We are not testing upgrade to far away version 100.3 to prevent this issue from happening for a long time: https://github.com/sparta-science/connect/commit/dbf99c78ea8f77049bcd08c65a89abb01a612a98


RFP - friction

Checklist

[X] The package I am requesting does not already exist on https://community.chocolatey.org/packages;
[X] There is no open issue for this package;
[x] The issue title starts with 'RFP - ';
[X] The download URL is public and not locked behind a paywall / login;

Package Details
Software project URL : https://friction.graphics
Direct download URL for the software / installer : https://github.com/friction2d/friction/releases/download/v0.9.4/friction-0.9.4-setup-win64.exe
Software summary / short description: an open-source, vector-based application designed for creating web animations and videos.

We could not detect that you have provided a URL to the software project site, please add this to the 'Software project URL:' part of the template.

https://friction.graphics/ is a valid URL

Is it on the same line?

Is it on the same line?

Indeed

I've had a look at the code, and unfortunately, I can't see where the issues lies. I'll leave that to @AdmiringWorm. In the meantime I'll change the labels.

The bot currently only thinks any URL that has the top-level domain of a maximum of 6 characters is valid. As the specified URL has an 8-character top-level domain, it isn't considered a valid one by the bot.

I thought the same and I amended this line to {1,10}. It may have been the wrong line, or I was holding it wrong, but it didn't make any difference.

I thought the same and I amended this line to {1,10}. It may have been the wrong line, or I was holding it wrong, but it didn't make any difference.

That is the line that needs to be updated, as well as line 59 as well.
I do not believe there is anywhere else, I'll look into it when I have time.

@pauby the top domain validation is now fixed and should allow top domains of up to 10 characters now.


Enhance Juriscraper to Support Bundling of Separate Opinions

Issue Description:
Currently, a handful of courts provide separate opinions in their opinion lists, which are not currently supported by juriscraper and CourtListener (CL). This lack of support for bundling separate opinions can lead to incomplete or segmented case information being scraped and processed.
Suggested Enhancement:
I propose updating juriscraper to allow for the bundling of separate opinions. This enhancement would ensure that all opinions related to a case are collected and processed together, providing a more comprehensive view of the case proceedings and decisions.
Courts: (in progress list)

[ ] Connecticut
[ ] Conn. Court of Appeals
[ ] West Virginia
[ ] West Virginia Court of Appeals
[ ] Michigan Court of Appeals
[ ] Tennessee Supreme Court
[ ] Texas Supreme
[ ] Texas Court of Appeals

To be clear here, what you're proposing is upgrading Juriscraper to return multiple opinion objects under one key, like we have with clusters/opinions in CL itself, right? Assuming so, can you provide a link or screenshot or something as an example?

Yes - I was working this thru in my head - before I laid out my vision.

{'date': '2/14/2023', 'docket': 'SC20164', 'name': 'State v. Juan A. G.-P.', 'opinion_type': '010combined'}

{'date': '1/31/2023', 'docket': 'SC20627', 'name': 'CT Freedom Alliance, LLC v. Dept. of Education', 'opinion_type': '010combined'}

{'date': '1/31/2023', 'docket': 'SC20633', 'name': 'Devine v. Fusaro', 'opinion_type': '010combined'}

{'date': '1/24/2023', 'docket': 'SC20679', 'name': 'Grant v. Commissioner of Correction', 'opinion_type': '010combined'}

{'date': '1/24/2023', 'docket': 'SC20371', 'name': 'State v. Brandon', 'opinion_type': '010combined'}
{'date': '1/24/2023', 'docket': 'SC20371', 'name': 'State v. Brandon', 'opinion_type': '030concurrence'}
{'date': '1/24/2023', 'docket': 'SC20371', 'name': 'State v. Brandon', 'opinion_type': '040dissent'}

{'date': '1/24/2023', 'docket': 'SC20597', 'name': 'Solon v. Slater', 'opinion_type': '010combined'}

{'date': '1/17/2023', 'docket': 'SC20453', 'name': 'State v. James A.', 'opinion_type': '010combined'}
{'date': '1/17/2023', 'docket': 'SC20453', 'name': 'State v. James A.', 'opinion_type': '030concurrence'}

I fixed and rewrote part of Connecticut - to take advantage of the opinion_type changes.  Here are some excepts from self.cases
We can take these results and either call a method to combine the multiple opinions here into clusters and only slightly modify CL to save each opinion together with the cluster

I'd expect this to mirror the fields in CL pretty closely. Why not do the joining in JS so that CL has a nice JSON object of clusters with nested opinions?

I checked the changes required on Courtlistener to support this new paradigm, while still supporting the legacy scrapers. I found the following:

We can return a nested object, but we must keep a minimal interface (dict keys) for compatibility with cl_scrape_opinions tasks of dup checking

{
    "Docket": {...},
    "OpinionCluster": {...},
    "Opinion": {...},
    "case_names": "",  # for site.hash dup checking
    "download_urls: "",  # for sha1 checking of url content
    "precedential_statuses": "" # for sha1 checking in case of nev
    "case_dates": "", # for sorting and dup checking,
}

Even if we return objects of the following shape we would have to return an item for each opinion (because of dup checking), causing a somewhat ugly duplication
{
    "OpinionCluster": {
             "Opinion": [
                     {...},
                     {...}
              ]
    }
}

Returning the objects ready to create means we must pass all required values.
This object approach allows greater flexibility to add more fields as we found them, without having to modify CL each time
The objects returned can be validated by a JSON Schema as discussed in #838

Here is a branch where I show the changes needed in CL, which turned out rather small. This is still a concept, would have to be tested and improved
https://github.com/freelawproject/courtlistener/compare/main...grossir:courtlistener:support_juriscraper_nested_objects?expand=1

Gianfranco, it's very OK to change CL as part of this, if it means making the interface better while hitting our design requirements. I'd rather do this now and have something we like instead of being stuck with half measures. Does that change your thinking about approach?

It took quite some time but I have a draft working on integration with Courtlistener (which will be another parallel PR)
First I will paste some nice screenshots, then I will dive into some problems and opportunities I found while working on this
Results
I used tex as a working scraper to test the new class. As a useful example, we have this recent Supreme Court case, which has a OpinionCluster of 3 opinions.
This is how the cluster looks:

How it currently looks on Courtlistener

Also, the scraper captures search_originating_court_information

and extra columns for our usual objects
Implementation details
It's better to look at the code, even if there is still pending work. I have written comments extensively.
https://github.com/freelawproject/juriscraper/pull/952
Besides the "code" code review, I will need some "data" code review, to see if I am using properly the nature_of_suit, cause,  opinion.type, etc fields
Of note, I found a way to keep tests of secondary/deferred page's examples.  For tex it was as simple as tweaking the href leading to the secondary page, so that it points to the precise example file.
Pending work
I still have a bunch of bugs to solve and tests to write for this to be mergeable

writing tests for the JSON Validator (I know it is currently not validating nested objects)
writing a custom type checker for python dates
support deferring attributes
adapting texcrimapp and texapp_* to the new tex class

Further work
There is a clear opportunity to scrape people_db objects, like Person, Party, Attorney, and to support them in cl_scrape_opinions. However, this would take more work and testing since lookups for this objects have to be used
Some bugs found on the way
Bugs on OpinionSite[Linear] integration with CL: Attributes that we can return but are never picked up in CL (defined on OpinionSite class)
            "dispositions",
            "causes",
            "divisions",
            "docket_attachment_numbers",
            "docket_document_numbers",
            "lower_courts",
            "lower_court_judges",
            "lower_court_numbers",

These are actually used on some sources, so we are not inserting data we do collect. For example, lower_courts is used in tenn, nev, ind, bap1, etc


absPathFromFileUri() does not return absolute path always.

The URI returned in the "results" depends on the method and the LSP Server.

This issue is related to my PR https://github.com/Andriamanitra/mlsp/pull/10 (that solves https://github.com/Andriamanitra/mlsp/issues/7) where i assumed that LSPs always returned absolute paths, but that is not the case.
Feel free to close this.

I'm unable to get lua-language-server to respond with relative paths to any requests. Could you provide instructions to reproduce the issue or a log of it happening? I suspect the bug is actually the Buf.Path in this snippet (should be AbsPath) but I would like to verify that it actually fixes the issue you encountered.
https://github.com/Andriamanitra/mlsp/blob/23249789ba4d926197d799e0b08855fa80e31dd9/main.lua#L454-L463
absPathFromFileUri() is only ever supposed to be called with file URI (string starting with "file://") which I have assumed are always absolute paths. The else branch in the implementation is just a dirty hack (that I should probably get rid of...) to get around LSP protocol's DocumentSymbol object (in textDocument/documentSymbol responses) not containing the file path.
https://github.com/Andriamanitra/mlsp/blob/23249789ba4d926197d799e0b08855fa80e31dd9/main.lua#L1152-L1159

absPathFromFileUri() is only ever supposed to be called with file URI (string starting with "file://") which I have assumed are always absolute paths.

Correct, but the LSP give me "main.lua" so the else branch was executed.
I change your log function to automatically return (log.txt too big to open, hangs micro with even only executing document-symbols.

Thanks, the log confirms my suspicions – lua-ls is returning DocumentSymbol[] instead of SymbolInformation[] so it ends up executing line 458 here which is where the relative path comes from:
https://github.com/Andriamanitra/mlsp/blob/23249789ba4d926197d799e0b08855fa80e31dd9/main.lua#L454-L463


feat: support npm resolution for file imports

Fixes #4098.
What this PR solves / how to test:
Previously when importing static files (such as wasm) from an npm package, you had to path directly to it, such as :
import wasm from '../../node_modules/svg2png-wasm/svg2png_wasm_bg.wasm'

While this works file, the DX isn't ideal, so this PR adds proper package resolution for these statements, allowing you to do:
import wasm from 'svg2png-wasm/svg2png_wasm_bg.wasm'

This uses resolve.exports to look at and analyse the package's exports field, and resolve to the applicable file. This does add an additional dependency, but this is a very common one used by so much build tooling including vite, jest, svelte, and more.
If for any reason this fails, it falls back to the previous behaviour and will just do a path reference, and then end up throwing an ENOENT as per previous functionality.
 It seems to work remarkebly well in practice across the many projects I've tested this against, but I'm happy to improve this if someone has a cleaner idea.
While I've added tests for the npm package name extraction to ensure it handles scoped packages, I've not added any specific tests for handling of the imports. To test that, it'd need a fixture with a real node_modules folder, and dummy package, which I don't think there's any prior art for in the repo today. Happy to add these if someone can guide me on the best way to approach this.

Author has included the following, where applicable:

[x] Tests
[x] Changeset (Changeset guidelines)

Reviewer is to perform the following, as applicable:

Checked for inclusion of relevant tests
Checked for inclusion of a relevant changeset
Checked for creation of associated docs updates
Manually pulled down the changes and spot-tested

Note for PR author:
We want to celebrate and highlight awesome PR review! If you think this PR received a particularly high-caliber review, please assign it the label highlight pr review so future reviewers can take inspiration and learn from it.

Can this be done more simply using the node.js require.resolve() function? For example something like:
let filePath = path.join(args.resolveDir, args.path);
if (!path.isAbsolute(args.path) && !args.path.startsWith(".")) {
	try {
	filePath = require.resolve(args.path)
} catch {
	// Do nothing
}

Note that your current checks for package like paths would break on Windows.

Can this be done more simply using the node.js require.resolve() function?

Unfoortunately not from my testing. Running require.resolve with args.path as svg2png-wasm/svg2png_wasm_bg.wasm results in Error: Cannot find module 'svg2png-wasm/svg2png_wasm_bg.wasm' with code MODULE_NOT_FOUND . I don't know exactly why this is, but I'd suspect it's something to do with CJS vs ESM interop, and newer exports mapping.

Note that your current checks for package like paths would break on Windows.

Can you expand here? My checks are purely checking for an included /, and not starting with ..
If I try and import like svg2png-wasm\\svg2png_wasm_bg.wasm, this plugin doesn't even run, and I get a generic esbuild error: No loader is configured for ".wasm" files: ../../../../example-wasm/node_modules/svg2png-wasm/svg2png_wasm_bg.wasm. I suspect this is because this plugin only runs on the specified filter, which looks something like this:
globs: ["**/*.wasm", "**/*.wasm?module"]

Supporting more windows orientated backslashes seems like a general issue here, and not really in the scope of this PR if I'm understanding correctly. It's also extremely uncommon, even on Windows from my experience, to import using backslashes in the path, especially with npm modules. This would also explain why no one has reported it as a bug.

Supporting more windows orientated backslashes seems like a general issue here, and not really in the scope of this PR if I'm understanding correctly. It's also extremely uncommon, even on Windows from my experience, to import using backslashes in the path, especially with npm modules. This would also explain why no one has reported it as a bug.

Yeah that is a good spot. The rules themselves wouldn't match in the first place.

All of the E2E test failures look unrelated, failing due to AssertionError: Please provide a CLOUDFLARE_ACCOUNT_ID as an environment variable

Thanks! I've added two test fixtures now - one just exporting a simple multiply.wasm file (stolen from another fixture), and then the other importing it. In doing so, I found some small issues with how I was using resolveDir vs process.cwd() so got that all fixed up.
Let me know if there's any further changes needed here.

The failing tests seem to be in external-service-bindings-app:test:ci, and unrelated to this PR.

Nevermind, something in service-bindings-app is still flaking and throwing a v8 error. Completely unrelated to this PR, but preventing it from going green.


Rewrite front-page to be more relevant to readers

:white_check_mark: Checklist

[x] This pull request has a meaningful title.
[ ] If your changes are not yet ready to merge, you have marked this pull request as a draft pull request.

:ballot_box_with_check: Maintainers' checklist

[x] This pull request has had the appropriate labels assigned
[x] This pull request has been added to the SATRE backlog project board
[ ] This pull request has been assigned to one or more maintainers

:arrow_heading_up: Summary
The front-page focusses on the background to SATRE. We should make the front-page more attractive to people, and get straight to the point.
This includes making the evaluation spreadsheet more obvious (feedback from a Collaboration Cafe)
:closed_umbrella: Related issues

:raising_hand: Acknowledging contributors

[x] All contributors to this pull request are already named in the table of contributors in the README file.
[ ] The following people should be added to the table of contributors in the README file:

After my first changes to the front page I realised there's no easy way to link directly to the content of the specification, so I've modified specification.md to be the entrypoint, and moved the background information out of it.


vagrant setup error

i got 2 errors on setup, Can anyone commit please. I am just getting going on Weave and want to make sure the base setup is ok.
The red error text ( pity github issues dont have colour !) is:

==> weave-gs-01: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.LjEo8jLdAb --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
==> weave-gs-01: gpg:
==> weave-gs-01: requesting key A88D21E9 from hkp server keyserver.ubuntu.com
==> weave-gs-01: gpg:
==> weave-gs-01: key A88D21E9: public key "Docker Release Tool (releasedocker<EMAIL_ADDRESS>imported
==> weave-gs-01: gpg:
==> weave-gs-01: Total number processed: 1
==> weave-gs-01: gpg:
==> weave-gs-01:               imported: 1
==> weave-gs-01:   (RSA: 1)

```

ENV:
golang: 1.5
vagrant: latest
virtbox: latest

---

Here we go:

cd ./guides/ubuntu-simple
vagrant up
Bringing machine 'weave-gs-01' up with 'virtualbox' provider...
Bringing machine 'weave-gs-02' up with 'virtualbox' provider...
==> weave-gs-01: Importing base box 'ubuntu/trusty64'...
==> weave-gs-01: Matching MAC address for NAT networking...
==> weave-gs-01: Checking if box 'ubuntu/trusty64' is up to date...
==> weave-gs-01: A newer version of the box 'ubuntu/trusty64' is available! You currently
==> weave-gs-01: have version '20151211.1.0'. The latest is version '20160120.0.0'. Run
==> weave-gs-01: vagrant box update to update.
==> weave-gs-01: Setting the name of the VM: ubuntu-simple_weave-gs-01_1453461645592_78604
==> weave-gs-01: Clearing any previously set forwarded ports...
==> weave-gs-01: Fixed port collision for 22 => 2222. Now on port 2200.
==> weave-gs-01: Clearing any previously set network interfaces...
==> weave-gs-01: Preparing network interfaces based on configuration...
weave-gs-01: Adapter 1: nat
weave-gs-01: Adapter 2: hostonly
==> weave-gs-01: Forwarding ports...
weave-gs-01: 22 => 2200 (adapter 1)
==> weave-gs-01: Running 'pre-boot' VM customizations...
==> weave-gs-01: Booting VM...
==> weave-gs-01: Waiting for machine to boot. This may take a few minutes...
weave-gs-01: SSH address: <IP_ADDRESS>:2200
weave-gs-01: SSH username: vagrant
weave-gs-01: SSH auth method: private key
weave-gs-01: Warning: Connection timeout. Retrying...
weave-gs-01:
weave-gs-01: Vagrant insecure key detected. Vagrant will automatically replace
weave-gs-01: this with a newly generated keypair for better security.
weave-gs-01:
weave-gs-01: Inserting generated public key within guest...
weave-gs-01: Removing insecure key from the guest if it's present...
weave-gs-01: Key inserted! Disconnecting and reconnecting using new SSH key...
==> weave-gs-01: Machine booted and ready!
==> weave-gs-01: Checking for guest additions in VM...
weave-gs-01: The guest additions on this VM do not match the installed version of
weave-gs-01: VirtualBox! In most cases this is fine, but in rare cases it can
weave-gs-01: prevent things such as shared folders from working properly. If you see
weave-gs-01: shared folder errors, please make sure the guest additions within the
weave-gs-01: virtual machine match the version of VirtualBox you have installed on
weave-gs-01: your host and reload your VM.
weave-gs-01:
weave-gs-01: Guest Additions Version: 4.3.34
weave-gs-01: VirtualBox Version: 5.0
==> weave-gs-01: Setting hostname...
==> weave-gs-01: Configuring and enabling network interfaces...
==> weave-gs-01: Mounting shared folders...
weave-gs-01: /vagrant => /Users/apple/workspace/go/src/bitbucket.org/apparently_different/backend-x/weave/guides/ubuntu-simple
==> weave-gs-01: Running provisioner: shell...
weave-gs-01: Running: inline script
==> weave-gs-01: stdin: is not a tty
==> weave-gs-01: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.IXXA68YF5o --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
==> weave-gs-01: gpg:
==> weave-gs-01: requesting key A88D21E9 from hkp server keyserver.ubuntu.com
==> weave-gs-01: gpg:
==> weave-gs-01: key A88D21E9: public key "Docker Release Tool (releasedocker<EMAIL_ADDRESS>imported
==> weave-gs-01: gpg:
==> weave-gs-01: Total number processed: 1
==> weave-gs-01: gpg:
==> weave-gs-01:               imported: 1
==> weave-gs-01:   (RSA: 1)
==> weave-gs-01: Selecting previously unselected package aufs-tools.
==> weave-gs-01: (Reading database ... 62972 files and directories currently installed.)
==> weave-gs-01: Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1_amd64.deb ...
==> weave-gs-01: Unpacking aufs-tools (1:3.2+20130722-1.1) ...
==> weave-gs-01: Selecting previously unselected package liberror-perl.
==> weave-gs-01: Preparing to unpack .../liberror-perl_0.17-1.1_all.deb ...
==> weave-gs-01: Unpacking liberror-perl (0.17-1.1) ...
==> weave-gs-01: Selecting previously unselected package git-man.
==> weave-gs-01: Preparing to unpack .../git-man_1%3a1.9.1-1ubuntu0.2_all.deb ...
==> weave-gs-01: Unpacking git-man (1:1.9.1-1ubuntu0.2) ...
==> weave-gs-01: Selecting previously unselected package git.
==> weave-gs-01: Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.2_amd64.deb ...
==> weave-gs-01: Unpacking git (1:1.9.1-1ubuntu0.2) ...
==> weave-gs-01: Selecting previously unselected package cgroup-lite.
==> weave-gs-01: Preparing to unpack .../cgroup-lite_1.9_all.deb ...
==> weave-gs-01: Unpacking cgroup-lite (1.9) ...
==> weave-gs-01: Selecting previously unselected package lxc-docker-1.9.1.
==> weave-gs-01: Preparing to unpack .../lxc-docker-1.9.1_1.9.1_amd64.deb ...
==> weave-gs-01: Unpacking lxc-docker-1.9.1 (1.9.1) ...
==> weave-gs-01: Selecting previously unselected package lxc-docker.
==> weave-gs-01: Preparing to unpack .../lxc-docker_1.9.1_amd64.deb ...
==> weave-gs-01: Unpacking lxc-docker (1.9.1) ...
==> weave-gs-01: Processing triggers for man-db (<IP_ADDRESS>-1ubuntu1) ...
==> weave-gs-01: Processing triggers for ureadahead (0.100.0-16) ...
==> weave-gs-01: Setting up aufs-tools (1:3.2+20130722-1.1) ...
==> weave-gs-01: Setting up liberror-perl (0.17-1.1) ...
==> weave-gs-01: Setting up git-man (1:1.9.1-1ubuntu0.2) ...
==> weave-gs-01: Setting up git (1:1.9.1-1ubuntu0.2) ...
==> weave-gs-01: Setting up cgroup-lite (1.9) ...
==> weave-gs-01: cgroup-lite start/running
==> weave-gs-01: Setting up lxc-docker-1.9.1 (1.9.1) ...
==> weave-gs-01: docker start/running, process 5072
==> weave-gs-01: Processing triggers for ureadahead (0.100.0-16) ...
==> weave-gs-01: Setting up lxc-docker (1.9.1) ...
==> weave-gs-01: Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
==> weave-gs-02: Importing base box 'ubuntu/trusty64'...
==> weave-gs-02: Matching MAC address for NAT networking...
==> weave-gs-02: Checking if box 'ubuntu/trusty64' is up to date...
==> weave-gs-02: A newer version of the box 'ubuntu/trusty64' is available! You currently
==> weave-gs-02: have version '20151211.1.0'. The latest is version '20160120.0.0'. Run
==> weave-gs-02: vagrant box update to update.
==> weave-gs-02: Setting the name of the VM: ubuntu-simple_weave-gs-02_1453461809933_60084
==> weave-gs-02: Clearing any previously set forwarded ports...
==> weave-gs-02: Fixed port collision for 22 => 2222. Now on port 2201.
==> weave-gs-02: Clearing any previously set network interfaces...
==> weave-gs-02: Preparing network interfaces based on configuration...
weave-gs-02: Adapter 1: nat
weave-gs-02: Adapter 2: hostonly
==> weave-gs-02: Forwarding ports...
weave-gs-02: 22 => 2201 (adapter 1)
==> weave-gs-02: Running 'pre-boot' VM customizations...
==> weave-gs-02: Booting VM...
==> weave-gs-02: Waiting for machine to boot. This may take a few minutes...
weave-gs-02: SSH address: <IP_ADDRESS>:2201
weave-gs-02: SSH username: vagrant
weave-gs-02: SSH auth method: private key
weave-gs-02: Warning: Connection timeout. Retrying...
weave-gs-02:
weave-gs-02: Vagrant insecure key detected. Vagrant will automatically replace
weave-gs-02: this with a newly generated keypair for better security.
weave-gs-02:
weave-gs-02: Inserting generated public key within guest...
weave-gs-02: Removing insecure key from the guest if it's present...
weave-gs-02: Key inserted! Disconnecting and reconnecting using new SSH key...
==> weave-gs-02: Machine booted and ready!
==> weave-gs-02: Checking for guest additions in VM...
weave-gs-02: The guest additions on this VM do not match the installed version of
weave-gs-02: VirtualBox! In most cases this is fine, but in rare cases it can
weave-gs-02: prevent things such as shared folders from working properly.
Current machine states:
weave-gs-01               running (virtualbox)
weave-gs-02               running (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run vagrant status NAME.
joe99-MacBook-Pro:weave apple$

Is fine. Just spurious...


BMA421 work in interrupt-driven mode

Now the bma421 driver could poll data from, but can not in interrupt-driven mode to save more power

Hi! Did you manage to get the hardware interrupt working?

I tried on my another watch which is bma423, everything is well, but I am not sure bma421 is not a public version chip, that is worth to try

It might be not possible on a hardware level on the BMA321,its product brief (http://static6.arrow.com/aropdfconversion/ad8392ae5860726c75b2a425eebd94416f860baa/25bma421_productflyer_bst_20170328.pdf) states:

Embedded Intelligence:
Step detector / Step counting
Any-motion/No-motion

whereas the BMA422 brief (https://www.bosch-sensortec.com/media/boschsensortec/downloads/product_flyer/bst-bma422-fl000.pdf) states:

Embedded Intelligence:
Step detector / Step counting
Significant motion
Tilt detection
Wake-up, Glance, Pick-up

and finally, the "wearables feature set" (https://www.bosch-sensortec.com/media/boschsensortec/downloads/application_notes_1/bst-mas-an032-00.pdf), which applies to "BMA423, BMA425, BMA456" supports

Single-tap
Step detector
Activity recognition
Wrist wear wakeup
Double-tap
Any-motion
No-motion
Error Interrupt

The driver code for the BMA421 in InfiniTime is based on the public BMA423 repo by Bosch. So I guess even though interrupts on the BMA421 could work in theory, the interrupt sources are rather limited. My main interest would be on tap detection, but that  seems to be unavailable in hardware.

I have look at interrupt bitmaskt is not same as bma423, so the interrupt source code maybe is not suitable for bma421, just need some modifications

I will try based drivers/sensor/bma421/libbma421/ to see the result, not sure the config file correct because is not the same with bma423 modified by InfiniTime

Thanks for the tip, I plan to compare your implementation against the InfiniTime one to see what differences there are in the register configurations. I can't say when I have time to continue, but I'll report back if I get it working.

I will reference on my bma423 code which is successfully work with interrupt driven same as Zephyr driver

The data ready interrupt work now, will try step counter later

Just my grain of salt in here:
BMA421 is a closed component. It needs a binary sequence to be uploaded during the initialisation to work properly. (At least to get the step counter working)
So Either you actually contact Bosch and sign a NDA with them to get the actual binary... Or you dig a bit the internet to find out that people already did a bit a retro engineering and got a binary that seems to work (I would look into the micro python FW of the pinetime ;) )
But Either way. You won't be allowed to upstream that driver to Zephyr..

@sdorre Bothe the micropython based wasp-os driver (https://github.com/daniel-thompson/bma42x-upy/blob/master/BMA42X-Sensor-API/bma421_config.h#L8=) as well as InfiniTime (https://github.com/InfiniTimeOrg/InfiniTime/blob/develop/src/drivers/Bma421_C/bma423.c#L561=) already use the reverse config blob (I suppose by https://gist.github.com/atc1441/788b5320050f50d65f71338e6e89c224), in order to get step counting working properly.
The question remains is what interrupt events the BMA421 is able to generate, and whether these events can be mapped to the actual hardware pin. I found various opinions on what the chips is exactly capable of, and the ability to generate interrupts might be enabled / disabled in the config blob.

now I am using  (https://github.com/daniel-thompson/bma42x-upy/blob/master/BMA42X-Sensor-API/bma421_config.h#L8=) it could generate data ready interrupt

Thank you for the testing, this helps me a lot. I still suspect that the BMA421 cannot detect complex motion events, e.g. double taps, on a hardware level.

No the BMP421 is limited to :

Step count
step detection
Any/No motions

Look at the datasheet
https://pdf1.alldatasheet.com/datasheet-pdf/view/1137426/BOSCH/BMA421.html
But the Touchscreen can do double tap ;-)

@sdorre Do you still have interesting in Zephyr pinetime firmware, I have almost finished the basic component, and need help!


missed_uplift: run only from mon to fri

Coverage decreased (-0.007%) to 27.918% when pulling<PHONE_NUMBER>ea7c8e5462bc3d83ef2a2f386e31f7 on calixteman:only_week into 4f7b84e55180be63c93efe5a294db5e4e0c50c9b on mozilla:master.


default value instead of np.nans for cooler matrix

Many algorithms need a matrix that has simply zeroes for missing bins, not NANs.
It would be nice to have a default value for c.matrix(balance=True)

can NaNs raise error messages?
I'm getting the following warning, not sure if it's caused by missing bins or something else:
WARNING:py.warnings:/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
WARNING:py.warnings:/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
WARNING:py.warnings:/usr/local/lib/python3.6/dist-packages/cooler/balance.py:355: RuntimeWarning: invalid value encountered in greater
logNzMarg = np.log(marg[marg>0])
WARNING:py.warnings:/usr/local/lib/python3.6/dist-packages/cooler/balance.py:359: RuntimeWarning: invalid value encountered in less
bias[marg < cutoff] = 0

Yes, NaNs can cause warnings like that. It looks like the marg array acquired at least one NaN value during the MAD-max filter step, where the marginals (row sums) are divided by their chromosomal median value.
My guess is that this is from 0/0 operations coming from chromosomes with no reads (like chrY or chrM), so it should be safe to ignore, assuming you are getting normal looking output.
We could probably avoid or suppress these warnings internally, though.


Im not able to installed

C:\Users\Rugved>npm install json-server
'npm' is not recognized as an internal or external command,
operable program or batch file.

I don't think that is a json-server problem. You can try to reinstall NodeJS, or apply fixes here.


XcodeServerEndpoints.endpointURL tests

I've created tests for endpointURL method from XcodeServerEndpoints class. I set access level of XcodeServerEndpoints.endpointURL method as internal as we discussed in #79 issue.
I also renamed EndPoints -> Endpoints in XcodeServerEndPoints name in order to maintain the compatibility between names of files, classes and variables.
Please review.

Result of Integration 1
Duration: 1 minute and 4 seconds
Result: Perfect build! All 56 tests passed. :+1:
Test Coverage: 54%.

This is some great stuff, @pmkowal, thanks so much! I added some comments, I'll have to look into the rev stuff. Also, why do you build up multiple params and test them all for the same thing, like in here: https://github.com/czechboy0/XcodeServerSDK/pull/80/files#diff-e671f822957fa3443397d33de2b2235eR105 ? Just curious :)

Thanks for comments @czechboy0 :smiley: The tests were modeled on the endpointURL method only. I know that there is a folder called routes in Xcode app package so I will also take a look at it :wink:.

Also, why do you build up multiple params and test them all for the same thing, like in here: https://github.com/czechboy0/XcodeServerSDK/pull/80/files#diff-e671f822957fa3443397d33de2b2235eR105 ? Just curious :)

I wanted to check all possibilites if additional/random parameters can affect the path creation.
Btw. In the meantime I will move a common sets of params to setUp() method.

Result of Integration 2
Duration: 48 seconds
Result: Perfect build! All 56 tests passed. :+1:
Test Coverage: 53%.

Btw. Weird thing about test coverage - I moved the common params from each method to setUp() method and test coverage changed from 54% to 53% - it looks like test coverage depends on the amount of code :smile:

Hmm yeah, I guess it's a percentage of tested code / all code? Including tests probably. Xcode 7 Code Coverage still needs some love I think :)

Ok, I made the code more explicit. See #82. I made the endpoints ONLY put in the rev when deleting a bot. This way you can remove a bit of the test code, where you're making sure the right values get put in when rev is in the params. Sorry that the code made it seem like rev was always needed, I guess I just never caught this case. Which just proves how crucial tests are here. Thanks again and sorry for the added work :blush:
Feel free to fix it whenever you have time :wink:

Ok, thanks! After I make fixes to the old tests I will merge #82 and improve tests :wink:
Btw. I'm currently checking routes and testEndpointURLCreationForBotsBotRevIntegrations test - is it possible to have this kind of path: /api/bots/:bot_id/:rev_id/integrations?

No, that's what I meant by guarding rev. Now rev ONLY gets put in if it's a DELETE request, which looks like DELETE /api/bots/:bot_id/:rev_id. Technically, it would be put in even if we had another DELETE request that used the bots endpoint as a prefix, but that's not the case right now so let's not worry about it. So you can remove all the mentions of rev apart from the one where we delete bot (which I added a test for in #82.)

Result of Integration 3
Duration: 48 seconds
Result: Perfect build! All 44 tests passed. :+1:
Test Coverage: 50%.

Result of Integration 4
Duration: 45 seconds
Result: Perfect build! All 45 tests passed. :+1:
Test Coverage: 51%.

@czechboy0 I made couple of fixes:

leaving tests which test success paths only
add let expectation to each test to make them more readable
"/api/bots/bot_id/rev_id" path uses rev only when method is DELETE

Please review.

Great, this is much better, thank you! (And just for the future, you don't have to bother with XCTAssertEqual's text as the third argument. The first two arguments are descriptive and when the test fails, the generated error basically says that they were not equal. Just to save you some time.)
So basically
XCTAssertEqual(url!, expectation, "endpointURL(.SCM_Branches) should return \(expectation)")

and
XCTAssertEqual(url!, expectation)

give you the same information, so you can just use the second one.

Thanks so much, @pmkowal! :+1:

Cool, thanks @czechboy0 !

👏
Officially, we've got a new contributor! 🎉

:fireworks:

:smile:


Inconsistent response inference for operation functions get vs get_with, post vs post_with, etc.

This issue tracks the tangential problem that was described in https://github.com/tamasfe/aide/issues/81#issuecomment-1733905157 after it was closed.
Using get (and analogous operation functions), response types are inferred, but using get_with does not infer the response types unless you globally enable them with infer_responses(true). Ideally, these operation functions would have the same inference behavior, or this could be better documented.
Would it be better if both used the global configuration (like get_with), or if both would automatically infer response types (like get)? Consolidating to either approach would be a breaking change.
On the one hand, I believe that the preferable default for both functions is to infer response types by default. Having the library infer the response type is the "I don't want to worry about it" approach that is consistent with not needing to explicitly configure anything. If you want manual control over the response types, that should be the explicit case.
On the other hand, if you have a global configuration that is supposed to decide whether responses should be inferred, then these functions should probably respect that, unless it is explicitly overridden in a narrower scope. Currently, the global default is false which means inference would be disabled for both of these, unless explicitly enabled.
One way to reconcile these conflicting priorities would be by changing the default global configuration to true.

It makes no sense to me to disable response inferrence by default. If infer response global option has no use we should remove it.
get_with should definitely infer responses by default.


Can we encode using flutter_ffmpeg inside an isolate-bloc?

Encoding with flutter_ffmpeg takes too much time and resources and generates Jank in the UI. So it would be very nice if we could run it in an isolate thread. Can we do it with the help of your library? If yes, is there a way to get progress callbacks from the thread? I want to show the user how much encoding is done. Normally flutter_ffmpeg gives this callback in its API.
Also, is there a straightforward way to allow this isolate-bloc to keep on working while the app is in the background?

Hi.
If you are using bloc for state management you may convert your bloc to the isolate bloc.
Or may create an isolate bloc and do your job into it.
However if you just want to convert image inside isolate you should use some libraries which helps you work with isolates or read something about them, they aren't so scary as you may think.

Unfortunately, only native services can works in background.


Re-enable 2FA metrics, speed up queries by denormalizing counts

In https://github.com/pypa/warehouse/pull/11699 we disabled the 2FA metrics temporarily due to DB performance issues.
We should improve the count performance with something similar to https://github.com/pypa/warehouse/pull/745/ and re-enable the metrics.

Create new column for User.has_webauthn, make it non-nullable, default false

Double check that we're on PG 11+, prior to PG 11 adding a column with a default wasn't safe, but in PG 11 it became safe.

To make sure this doesn't maintain an exclusive lock on the db, commit and open transactions in upgrade method, similar to

Just make sure that you chunk the data backfill, and that your update query ignores things that have already been migrated. That may be easier to do with 3 migrations:

Add the column, nullable no default + triggers.
Update batches, with a WHERE has_webauthn IS NULL limitation.
Set the column to not nullable, default of false.

I also just looked, our env.py for alembic contains these lines:
context.configure(
            connection=connection,
            target_metadata=db.metadata,
            compare_server_default=True,
        )
with context.begin_transaction():
            context.run_migrations()

In the run_migrations_online() method. It appears by default alembic will open a single transaction and run all of the migrations inside of that. I think we want to change that to:
context.configure(
            connection=connection,
            target_metadata=db.metadata,
            compare_server_default=True,
            transaction_per_migration=True,
        )
with context.begin_transaction():
            context.run_migrations()

This will make the transaction opened by context.begin_transaction a logical transaction (rather than a database transaction), and internally inside of context.run_migrations() before executing an upgrade() method, alembic will open up a real database transaction.
I also see that there's actually a mode to let you turn on PG's auto commit inside of a migration:
def upgrade():
    # Every statement inside of the autocommit block, will be automatically committed.
    with op.get_context().autocommit_block():
        # We need to use get_bind() here because op.execute doesn't return a result.
        conn = op.get_bind()

        # Update users in chunks, until no more users have a null has_webauthn
        while True:
            r = conn.execute("UPDATE ... WHERE id IN (SELECT id FROM users WHERE has_webauthn IS NULL LIMIT 1000)")
            if r.rowcount == 0:
                break

Update task/expression to consider User.has_webauthn instead

You can refactor the User.has_two_factor to be something like:
@hybrid_property
    def has_two_factor(self):
        return self.totp_secret is not None or self.has_webauthn

 @has_two_factor.expression
 def has_two_factor(self):
    return or_(self.totp_secret_is_not(None), self.has_webauthn)

Then just continue to use User.has_two_factor, since it'll just be using two columns on the user table, it won't involve any gnarly sub selects, and should be fast. Might be able to speed it up with an index, not sure.

Also to be clear, I didn't actually EXPLAIN that query to figure out what the problem was, I'm just eyeballing it and guessing. It might be useful to do EXPLAIN or EXPLAIN ANAYLIZE on it and make sure that my assumption is correct... but I'm pretty sure it is.  I can't imagine that subquery performing well.

The local dev database obtained with make initdb shows a total user count of 40,239. Is this demonstrative of the same user count in production, or are there more?
Again, in development: The query performs two sequential scans, one on users and one on user_security_keys (0 in dev) - so if either of those tables are largish, there may be a good partial index opportunity on users.totp_secret.

warehouse=> SELECT COUNT(*) FROM users;
 count
--------
 604282
(1 row)

warehouse=> SELECT COUNT(*) FROM user_security_keys;
 count
-------
  4210
(1 row)

 Aggregate  (cost=6127375.34..6127375.35 rows=1 width=8) (actual time=346089.288..346089.292 rows=1 loops=1)
   ->  Seq Scan on users  (cost=0.00..6126620.24 rows=302040 width=0) (actual time=1.329..346068.003 rows=27481 loops=1)
         Filter: (SubPlan 1)
         Rows Removed by Filter: 576801
         SubPlan 1
           ->  Limit  (cost=0.00..10.11 rows=1 width=1) (actual time=0.571..0.571 rows=0 loops=604282)
                 ->  Seq Scan on user_security_keys  (cost=0.00..222.34 rows=22 width=1) (actual time=0.569..0.569 rows=
0 loops=604282)
                       Filter: ((user_id = users.id) OR ((users.id = users.id) AND (users.totp_secret IS NOT NULL)))
                       Rows Removed by Filter: 4024
 Planning Time: 0.121 ms
 Execution Time: 346089.324 ms
(11 rows)

In this case, that one query was taking almost 6 minutes to run, there were other queries, and the task was running every 5 minutes.

Just because I nerd sniped myself into seeing if I could optimize that query without denormalization:
EXPLAIN ANALYZE SELECT COUNT(*)
FROM users
LEFT JOIN LATERAL (
    SELECT user_security_keys.id
    FROM user_security_keys
    WHERE user_security_keys.user_id = users.id
    FETCH FIRST 1 ROW ONLY
) webauthn ON true
WHERE users.totp_secret IS NOT NULL
    OR webauthn.id IS NOT NULL;

 Aggregate  (cost=5048886.60..5048886.61 rows=1 width=8) (actual time=1862.242..1862.246 rows=1 loops=1)
   ->  Nested Loop Left Join  (cost=0.28..5047376.40 rows=604080 width=0) (actual time=0.045..1852.948 rows=27481 loops=1)
         Filter: ((users.totp_secret IS NOT NULL) OR (user_security_keys.id IS NOT NULL))
         Rows Removed by Filter: 576817
         ->  Seq Scan on users  (cost=0.00..21430.80 rows=604080 width=37) (actual time=0.011..295.354 rows=604298 loops=1)
         ->  Limit  (cost=0.28..8.30 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=604298)
               ->  Index Scan using ix_user_security_keys_user_id on user_security_keys  (cost=0.28..8.30 rows=1 width=16) (actual time=0.001..0.001 rows=0 loops=604298)
                     Index Cond: (user_id = users.id)
 Planning Time: 0.182 ms
 Execution Time: 1862.356 ms

346089ms down to 1862ms with the same results, no denormalization, no schema changes at all.

Nice! From this analyze I would hazard that the added index on users.totp_secret would reduce that sequential scan even further - interestingly webauthn.id makes no appearance.

I don't think it would, because the filter on users.totp_secret happens outside of the SeqScan, and logically you have to join all the rows to find out if there is webauthn attached to the user or not.

However, maintaining indexes isn't free, and either 1862ms or 326ms in a background task every 5 minutes is more than fast enough.

We ended up not doing most of this, and just optimizing the queries with joins instead.


Bug: Lambda is wrapping in the middle of a quoted string

Description:

It doesn't happen often, but the lambda code is wrapping in the middle of a quote and the resulting lines can no longer be compiled. This can be seen in the ttCode DlgExcludeBase dialog when the line is pasted in, saved, and reloaded.

I can't reproduce this. I suspect that since this issue was created, the styled text control is now set to automatically wrap. Unless wxUI wraps the text returned from the event editor (which is shouldn't) this should work fine.


API Exploits | Possibilty for XSS | Diabled API w/ No Keys = FreeAPI

Just one question appart from what the vedio makes apparent. Why are the API responses in plaintext?
@donaldzou

https://github.com/user-attachments/assets/d7f78e48-6653-43db-8172-cd30dbf89f0d

Hi @NOXCIS, I'm not too sure what issue the video is showing. Regarding possibility of XSS, I'm not an expert but CORS is disabled when there's no API key provided in headers, or I'm missing something else?

@donaldzou @DaanSelen
The dashboard allows unauthorized API calls while logged out after logging in the first time, even after clearing cookies and loacl data. A threat actor such as myslef,  would be able to leverage Cross Site Scripting by getting a user to load a malicious webpage that could then execute API calls to the dashboard if it were open in another tab. Hence dummbing the the wireguard config info without having to login.
I discovered this by accident while playing with flask request content to enable logging on my static binary version of the dashboard over on Wiregate. The loggin request were dumbing all the sensity data if the request content was also called.  Curiosity got the better of me and i tested the theroy on WgDashboard. To my horror it worked

// Function to call the Flask API endpoint
async function getWireguardConfigurations() {
    try {
        const response = await fetch('/api/getWireguardConfigurations', {
            method: 'GET', // Method itself
            headers: {
                'Content-Type': 'application/json' // Adjust as needed depending on your API response type
            }
        });

        // Check if the response is ok (status 200)
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }

        // Parse JSON response
        const data = await response.json();
        console.log('WireGuard Configurations:', data);
    } catch (error) {
        console.error('Error fetching WireGuard configurations:', error);
    }
}

// Call the function
getWireguardConfigurations();

Ah I see what you mean.. Yes, I will need to fix the issue where cookie is still exist after signing out. Regarding cross site scripting, I couldn't (I'm no expert) recreate a scenario where it could be vulnerable in WGDashboard's use case.. do you have any example?

Ah I see what you mean.. Yes, I will need to fix the issue where cookie is still exist after signing out. Regarding cross site scripting, I couldn't (I'm no expert) recreate a scenario where it could be vulnerable in WGDashboard's use case.. do you have any example?

@donaldzou For legal and public safety reasons I cannot provide a XXS example. I'll test later, but the fastest fix for updated users would be to enable at least one API key. Maybe after confirming a fix I can do a demo.

Sure! Please take your time :)
On Nov 25, 2024, at 2:23 AM, Noxcis @.***> wrote:
Ah I see what you mean.. Yes, I will need to fix the issue where cookie is still exist after signing out. Regarding cross site scripting, I couldn't (I'm no expert) recreate a scenario where it could be vulnerable in WGDashboard's use case.. do you have any example?
@donaldzouhttps://github.com/donaldzou For legal and public safety reasons I cannot provide a XXS example. I'll test later, but the fastest fix for updated users would be to enable at least one API key. Maybe after confirming a fix I can do a demo.
—
Reply to this email directly, view it on GitHubhttps://github.com/donaldzou/WGDashboard/issues/516#issuecomment-2496155213, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGARNUILSO3ZTSQPCSNL7432CIKQVAVCNFSM6AAAAABSIWZV6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJWGE2TKMRRGM.
You are receiving this because you were mentioned.Message ID: @.***>

Thanks @NOXCIS for your work. Good find!

@donaldzou merge into main? Because it's a security patch? Or what is your plan.

@donaldzou merge into main? Because it's a security patch? Or what is your plan.

@DaanSelen Yesss as I think this is important and planning to draft a release so people know to update to this fix ;) What do u think?

Yeah true, there should be like a patch release, like WGDashboard 4.1.1. The containers will only need to be redeployed because the WGDashboard code is stateless (If I am using the right term).

Yep! I think Docker should be all good 😊


Deployment Guide: Heroku

Docs PR: https://github.com/prisma/docs/pull/624
Example PR: https://github.com/prisma/prisma-examples/pull/1847

Closing. Both PRs have been merged


Feature Request: Import Images as individual Journals

Right now you can only mass import Images from a folder as either pages in a Journal or all images into one single text page.
It would be good to have the option to create a single Journal per image (with NameNNN for the structure, where NNN starts at 000 and it increases by +1 for each iteration).
This will allow the user to create a rollable table from the imported images - right now it is not possible as individual pages can't even be dragged into a Rollable table to use as content.

Right now you can only mass import Images from a folder as either pages in a Journal or all images into one single text page.
It would be good to have the option to create a single Journal per image (with NameNNN for the structure, where NNN starts at 000 and it increases by +1 for each iteration).
This will allow the user to create a rollable table from the imported images - right now it is not possible as individual pages can't even be dragged into a Rollable table to use as content.

I can take a look, but the page to table is a better solution
https://github.com/foundryvtt/foundryvtt/issues/9088

done. update

I can take a look, but the page to table is a better solution foundryvtt/foundryvtt#9088

Thanks for the super quick update! Dragging a page to the rollable table would unfortunately not work out well when you have to generate a large rollable table, like 100+ items.
I've just tried your solution and it works like a charm! I can:

Create a folder with 100+ Journal entries from images with this module;
Export the folder structure to a Journal Compendium (with the Compendium Folder module);
Create a rollable table from the folder inside the Compendium;
Delete the folder with the 100+ Journal entries from the sidebar
And now I have a weightless random image generator with 100+ entries :)