Do not show extra trains that go via airport [Feature request]

Would it make sense to filter out trains that run on the ring track via airport for the ring track destinations that can be reached directly? E.g. for Helsinki-Tapanila, we get now P, I and N trains where I and N trains take 14 minutes for the journey and P trains 45 minutes (see attached image). Usually, it does not make any sense to take the P train, since it costs more and is slower.
The same applies for all ring track destinations besides Pasila and Airport.

Actually, I tried to create the filter by modifying ringTrackFilterApplies, but could not quite do it (yet). I've never done Elm before. :-D It would be nice to do this change as pair coding though.

This is of course my personal opinion but after #21 got merged, I started to find user interface a bit more difficult to read while I'm on the go. What initially drew my attention to this app was simplicity of it.
I would consider of having some sort of rollback and instead just sort trains by their arrival time on the destination station. Similarly as the itinerary suggestions of HSLdevcom/digitransit-ui.

What precisely do you find more difficult than before? Maybe this can be made better.
Also, could you in the future try and empathize a little bit more with the people who make free apps or libraries for you to enjoy. "What initially drew my attention..." and "I would consider ... rollback" can be seen as a rather hostile way to communicate your point of view and I have to say I did (and do) feel bad after reading the comment.

@petja Thanks for filling in a potential solution that I forgot to consider! Sorting by arrival time at destination would help my use case 1 and perhaps 3 as well without making 2 impossible indeed, but on the other hand it wouldn't be a clean solution to any of these. For example, in use case 1, how would you find the second best train if you couldn't make it to the first one?
However, now that I think of it, perhaps the setting in #21 could be replaced by dimming the trains that are going to be overtaken by a later one? Then it would be up to you to take one anyway (use case 2) or skip them (use case 1 & 3). This could be considered as a variant of solution 3.

@ohanhi Sorry for my bad word choices and feeling they produced. By saying "what initially drew my attention" I meant to just open the discussion and tell what made me to use the app on the first place, but probably using past tense made comment look like I wouldn't use app anymore – which is of course not the case. "Rollback" was purely bad, sorry for that again. I'm taking my lesson. Thanks for pointing out these, I didn't see these as hostile when I wrote them.
I'm not saying app is now difficult to use, but more like it's not as easy as before. Moving metadata to the top was probably my issue. And the bars under it could be unclear to users who haven't read this thread.

Let me shed some light into my thoughts:

My own stations are such that all trains are roughly the same, so none of this data matters to me personally. I prefer not to see unnecessary data.
Kehärata is different from normal tracks. People commuting from e.g. Tapanila or Malminkartano to Pasila or Helsinki do not expect or want to see the trains that go the "long way". For them, trains via the Airport can be considered unnecessary data.
On normal tracks where a slower train is overtaken by a faster one, it depends on the use case what the user might want to see. I don't like the idea of sorting by arrival time, since I think of this as a departure station schedule screen (that only shows trains I could take). Also, dimming would reduce visual contrast and thus make it harder for low vision users to use the app.

Taking all this into account, I think I will remove the duration bars and numbers. I am visioning a notification text, shown only if the train would be overtaken by a later one.
Just to make it clear, I am not looking to support all use cases in an optimal way. A half-way solution (like the one I made) is likely to be worse for everyone, and adding more and more settings is not a road I want to take either.

Well reasoned. Sorting by arrival time has its problems if you're looking for trains departuring later than now, indeed.
Also, I really appreciate you want to keep things high contrast and thus respect people with low vision. Unfortunately this isn't prioritized thing at all on the most of the apps yet in 2019.


numpy array issue

I am having issues using the 'data' from the product object dict.
I am new to numpy, so I don't even know where to start looking for information. :(
Can you tell me how I can convert the numpy arrays into something I can use? I would love to be able to use the data as a pandas DataFrame, or even a list but it's coming across as a string & I know that there's a better option than building a parser to make it into a list.
Thanks!

If you provide a code example I can show you where to start.

This came about because I am trying to save the dictionary to MongoDB, which gave me the following error:
InvalidDocument: Cannot encode object: array([
       datetime.datetime(2017, 5, 25, 2, 32),
       datetime.datetime(2017, 5, 26, 10, 56),
       datetime.datetime(2017, 5, 27, 5, 32) ...

It doesn't like the numpy array.
To get around it I have removed the data key.
import` pymongo
import keepaAPI

access_key = 'xxxxxxxxxxxxxxxxxxxx'
api = keepaAPI.API(access_key)

client = pymongo.MongoClient(xxxxxxxxxxxxxxxxxxxx')
mdb = client.cuufan

p = api.ProductQuery('B0714KDKXW')[0]
del p['data']

mdb.keepa.insert_one(p)

which works, but the csv section of the requests is still a pain to work with and I feel like the numpy data will be more helpful if I know what to do with it.

It's not actually the numpy array, but the datetime object.  Looks like mongo just doesn't know how to work with datetime arrays, so you can disable this by setting to_datetime=False in ProductQuery.  See help(api.ProductQuery):
        """
        Performs a product query of a list, array, or single ASIN.  Returns a
        list of product data with one entry for each product.

        Parameters
        ----------
        asins : string, list, np.ndarray
            A list, array, or single ASIN.  Each ASIN should be 10
            characters and match a product on Amazon.  ASINs not matching
            Amazon product or duplicate ASINs will return no data.

        stats : int or date, optional
            No extra token cost. If specified the product object will have a
            stats field with quick access to current prices, min/max prices
            and the weighted mean values. If the offers parameter was used it
            will also provide stock counts and buy box information.

            You can provide the stats parameter in two forms:

            Last x days (positive integer value): calculates the stats of
            the last x days, where x is the value of the stats parameter.
            Interval: You can provide a date range for the stats
            calculation. You can specify the range via two timestamps
            (unix epoch time milliseconds) or two date strings (ISO8601,
            with or without time in UTC).

        domain : string, optional
            One of the following Amazon domains:
            RESERVED, US, GB, DE, FR, JP, CA, CN, IT, ES, IN, MX
            Defaults to US.

        offers : int, optional
            Adds available offers to product data.  Default 0.  Must be between
            20 and 100.

        update : int, optional
            if data is older than the input interger, keepa will update
            their database and return live data.  If set to 0 (live data),
            request may cost an additional token.  Default None

        history : bool, optional
            When set to True includes the price, sales, and offer history
            of a product.  Set to False to reduce request time if data is
            not required.  Default True

        rating : bool, optional
            When set to to True, includes the existing RATING and COUNT_REVIEWS
            history of the csv field.  Default False

        nthreads : int, optional
            Number of threads to interface to keepa with.  More threads
            means potentially faster batch response, but more bandwidth.
            Probably should be kept under 20.  Default 4

        to_datetime : bool, optional
            Modifies numpy minutes to datetime.datetime values.  Default True.

        allow_errors : bool, optional
            Permits errors in requests.  List of products may not match input
            asin list.

        out_of_stock_as_nan : bool, optional
            When True, prices are NAN when price category is out of stock.
            When False, prices are -0.01
            Default True

        Returns
        -------
        products : list

            See: https://keepa.com/#!discuss/t/product-object/116

            List of products.  Each product within the list is a dictionary.
            The keys of each item may vary, so see the keys within each product
            for further details.

            Each product should contain at a minimum a "data" key containing
            a formatted dictonary with the following fields:

        Notes
        -----
        The following are data fields a product dictionary

I will try that. Thanks so much for your help.

Sorry, I'm back again.
p = api.ProductQuery('B01KYA6SXI', to_datetime=False)[0]
mdb.testing.insert_one(p)

Produces this error:
InvalidDocument: Cannot encode object: array(['2017-08-13T05:50'], dtype='datetime64[m]')

Just use astype(‘int’) on the offending array


Fix typos

Checklist

[x] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[x] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[x] I've read the Contribution Guidelines
[x] I've updated the documentation if necessary.

Motivation and Context
This change fixes some typos in test names and a message.
Description
Fix misspellings of "environment" and "exist" and change "that not exist" to "that do not exist".
I ran bundle exec fastlane test, bundle exec rspec, and bundle exec rubocop -a.

I've signed the CLA.  Would you please re-scan?

Attempt to trigger Google CLA: @googlebot I signed it!

@joshdholtz it seems like Google CLA bot isn't posting instructions on PRs anymore 🙇

I noticed that yesterday 😩 I'll add them to the PR description and adjust fastlane bot I think if I can't get hold of someone from Google to fix that 😬


Bump default etcd version to v3.3.10

Kubeadm v1.14 requires an etcd version greater than v3.2.18.
(https://github.com/kubernetes/kubernetes/blob/v1.14.0/cmd/kubeadm/app/constants/constants.go#L257)
Bump the default etcd version to v3.3.10, which is the default version
used in the Kubernetes v1.14 release.
(https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#external-dependencies)
Fixes #178
Signed-off-by: Alexander Brand<EMAIL_ADDRESS>

I would like for us to explore how we can get this migrated to be kubeadm managed.  We took a path with systemd as it was the only viable option at the time. It might be time to re-explore this.

@craigtracey That sounds great to me. I'll close this PR and open an issue to track that migration.


regression on handling of valid multipolygons

Commit 3ad4e297983a8b08dc1653d0e40aeb75a361419d has broken the handling of valid inner rings in new-style multipolygons. Consider this simple test file that has a forest with a hole consisting of farmland:
<osm version="0.6">
 <node id="1" lat="10" lon="10"/>
 <node id="2" lat="10" lon="0"/>
 <node id="3" lat="0" lon="0"/>
 <node id="4" lat="0" lon="10"/>
 <node id="11" lat="5" lon="5"/>
 <node id="12" lat="5" lon="4"/>
 <node id="13" lat="4" lon="4"/>
 <node id="14" lat="4" lon="5"/>
 <way id="1">
  <nd ref="1"/>
  <nd ref="2"/>
  <nd ref="3"/>
  <nd ref="4"/>
  <nd ref="1"/>
 </way>
 <way id="2">
  <nd ref="11"/>
  <nd ref="12"/>
  <nd ref="13"/>
  <nd ref="14"/>
  <nd ref="11"/>
  <tag k="landuse" v="farmland"/>
 </way>
 <relation id="1">
  <member type="way" ref="1" role="outer"/>
  <member type="way" ref="2" role="inner"/>
  <tag k="natural" v="wood"/>
  <tag k="type" v="multipolygon"/>
 </relation>
</osm>

The expected output of this, when run through e.g. osmium_area_test -o, is two polygons - a forest with a hole, and a farmland. Like this (slightly shortened):
AREA: [240]
  id=3
  TAGS: [21]
    k=|natural| v=|wood|
  OUTER RING: [88]
    ref=3 pos=(0,0)
    ref=4 pos=(10,0)
    ref=1 pos=(10,10)
    ref=2 pos=(0,10)
    ref=3 pos=(0,0)
  INNER RING: [88]
    ref=13 pos=(4,4)
    ref=12 pos=(4,5)
    ref=11 pos=(5,5)
    ref=14 pos=(5,4)
    ref=13 pos=(4,4)
AREA: [160]
  id=4
  TAGS: [25]
    k=|landuse| v=|farmland|
  OUTER RING: [88]
    ref=13 pos=(4,4)
    ref=14 pos=(5,4)
    ref=11 pos=(5,5)
    ref=12 pos=(4,5)
    ref=13 pos=(4,4)

This works with all versions prior to the commit. After, and up until the current HEAD version at the time of submitting this ticket, we have this output (again, slightly shortened):
AREA: [240]
  id=3
  TAGS: [21]
    k=|natural| v=|wood|
  OUTER RING: [88]
    ref=3 pos=(0,0)
    ref=4 pos=(10,0)
    ref=1 pos=(10,10)
    ref=2 pos=(0,10)
    ref=3 pos=(0,0)
  INNER RING: [88]
    ref=13 pos=(4,4)
    ref=12 pos=(4,5)
    ref=11 pos=(5,5)
    ref=14 pos=(5,4)
    ref=13 pos=(4,4)

i.e. the separate farmland polygon is simply missing.

The osmium_area_test program from the examples that you are using has a built-in filter for areas tagged landuse=forest or natural=wood only. This is why the closed way tagged landuse=farmland doesn't appear in its output. If you use https://github.com/osmcode/osm-area-tools/blob/master/src/oat_create_areas.cpp for instance, you'll see it appear in the output.

Sorry for the confusion. I changed the inner ring to wood and the outer to farmland, and sure enough, the inner ring is output, so the bug I thought I saw is a phantom. I still wonder why, since the landuse=forest/natural=wood filter existed in osmium_area_test even before the commit in question, the older builds did output the farmland though.


[fpga] Modify and add new placement rules for BRAM

The main change of this PR is the modification and addition of new BRAM placement rules. Previously, we just used placement rules to enable Boot ROM splicing. But the placement was rather arbitrary, used some sites not available on smaller FPGAs and the checking was incomplete. This is fixed by this PR.
In addition, new placement rules for all non-Boot ROM BRAMs are added for the CW310 target. This is to guide Vivado P&R to:

Speed up implementation times, this is especially relevant for CI.
Reduce variation between implementation runs, this is relevant for improved reproducibility of SCA experiments.

As the design continues to evolve, we might need to adjust these rules from time to time. The required commands for extracting these rules from an existing design are now documented in the placement_cw310.xdc file.
Finally, a set_max_delay constraint is added to reduce variation in the SCA trigger delay between implementation runs.

Thanks Michael for your review. Haven't had the time yet to perform the experiments, it's time consuming. But will look into this, too.
In the meantime I did another rebase and adjust the placement rules. I looks like specifing LOC constraints for all BRAMs doesn't really help to speed up implementation. Maybe it's overconstrained. Retrying again with specifying just a few BRAMs. This gave good results initially.

Hi @imphil and @msfschaffner thanks for your reviews. The latest results are pretty disheartening. I did four runs starting from the exact same sources:

A first run without placement constraints to get a working placement.
A second run specifying all BRAM locations as in the previous run.
A third and fourth run where only few BRAM locations were specified.

There was zero improvement for the implementation time. Even worse, the last runs with only few BRAM locations specified took significantly longer (+1 or even 2 hours).
Also from an SCA perspective things are not looking good.

Hm, ok if this does not improve synthesis time, does it make sense to add BRAM constraints at all then?
For the SCA measurements, are you saying that the timing constraint on the trigger signal did not help at all?
Side note RE variability of synthesis runs: It looks like Vivado does not offer a synthesis P&R seed setting anymore, and hence it should produce identical results when build from the same sources and with the same synthesis strategy. However, results may be pretty different if something in the design changes (see this post here: https://forums.xilinx.com/t5/Implementation/Multiple-Seed-Place-amp-route/td-p/503024).

Hm, ok if this does not improve synthesis time, does it make sense to add BRAM constraints at all then?

Yes, I agree. We shouldn't add them if they are of no use.

For the SCA measurements, are you saying that the timing constraint on the trigger signal did not help at all?

Yep, that's correct.

Side note RE variability of synthesis runs: It looks like Vivado does not offer a synthesis P&R seed setting anymore, and hence it should produce identical results when built from the same sources and with the same synthesis strategy. However, results may be pretty different if something in the design changes (see this post here: https://forums.xilinx.com/t5/Implementation/Multiple-Seed-Place-amp-route/td-p/503024).

I saw that, too. Changing some BRAM location constraints is expected to lead to different results. This has also been suggested in the forum as a replacement for the no longer available seed setting. But I am really surprised to see the big differences in runtime when fixing just a few BRAMs compared to all.
Maybe I should also have a look at some optimization settings in the tool.

I am closing this PR as congestion issue seems to have been solved with #8138 and the BRAM placement didn't have a positive effect on FPGA build time or SCA reproducibility.


Remove docker image

moved scripts into CWL tasks; additional docker image not required anymore

Abandoned in favor of #5. Might be revisited after adding support for writable inputs in https://github.com/ohsu-comp-bio/cwl-tes


Work on reset password functionality.

Use bcrypt js for encrypting the password. Use nodemailer to send the reset password email. On clicking the link user will have the option to enter new password and confirm password. Encrypt the password and save it to database. Let the user schema be created first then work on this

I am working on the user schema,so i can implement it.

One participant has already told me that he wants to work on this issue. So I won't be able to assign you this issue. Don't worry I will assign some other issue

I will contribute to the reset-password functionality!


Rebuild jmespath 0.9.2 as noarch

Needed by downstream by globus-cli, which currently doesn't work with python 3.7

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@pmlandwehr, if you are willing to add me as a maintainer, I will immediately revert these changes.  If you prefer, you can create a new branch (e.g. 0.9.2) on conda-forge/jmespath-feedstock and change the destination of this PR to that branch.  In the latter case, after merging the PR, the packages will be built and uploaded but the recipe will remain unchanged in master (and I will not be added as a maintainer, which is fine).

Hey @xylar, I can certainly add you as a maintainer. However, the standard way to handle updating old versions of packages is by creating a version-specific fork and then making the change. I'll take care of that here.

@xylar and thanks for your attention!

@pmlandwehr, thanks for creating the v0.9.2 branch.  That's exactly what I meant.  It would have been possible to change the base of this PR to be the v0.9.2 branch and then merge, but what you did is fine, too.
Closing this PR because it has already been merged implicitly into v0.9.2


Add flake.nix-based reproducible builds

Building in nix develop works, but nix build -L ... is currently broken due to https://github.com/ipetkov/crane/issues/111

this is exciting, what would be a good way for me to test this out?

I wrote a README and then forgot to commit it. :facepalm: . See if it helps.

I might have some improvements soon, so let's hold with landing it.

OK. Made it work more properly (AFAICT) and added support for C++. This also works with compiling rocksdb, which will be included in another PR by @justinmoon

https://github.com/rust-rocksdb/rust-rocksdb/pull/682 breaks rocksdb on 32bit archs

Just to avoid confusion: I prepared this change on top of rocksdb branch which enabled rocksdb. I do not include that change in this PR though, to keep these things separate.

nix develop fails on m1 Mac with error: No Android SDK tarballs are available for system architecture: aarch64-darwin

@justinmoon Do we want to land it?

Looks like I forgot to comment here.
AFAICT this doesn't really help because rust code is built as a side-effect of running flutter run or flutter build via http://cjycode.com/flutter_rust_bridge/. I think it would probably be better to have the flake.nix in the root of the project for this reason. Let me revisit this today and try to get it working.


Request for Vue

Hey all. Is there any timeline on when this library will be available for Vue 3? I see it being referenced in the README in several places, but the Viewer seems to be only available for React.

The Vue 3 version of Viewer is under development and will be released as a beta version within some days.
It will be separated into another package because it's not common to bundle both React and Vue components in a single package.

Hey all, any news on this issue?

Here's an experimental Vue version: https://github.com/RexSkz/json-diff-kit-vue, please have a try.
Since I'm not too familiar with Vue 3, there may be some bugs in the component or the build system, but currently I may not have enough time to review & fix them :(


[jax2tf] Fix higher-order differentiation.

We must ensure that we call jax2tf.convert recursively to ensure that the proper tf.custom_gradient is used. This means that we can reuse the conversion of the VJP function between native and graph serialization.

@maxwillzq PTAL

@maxwillzq I had to make a few more changes to fix tests.
Note that there is a disabled test now in call_tf_test for non-native serialization, when attempting to save a model that was reloaded (convert -> save -> call_tf -> convert -> save). Since this occurs only with non-native serialization and is a corner case, I am fine disabling the test. If you do understand what is going on and can fix this, it would be best.

Got it.  Thanks a lot.


The model modifier cannot get the correct type?

Laravel Version: 5.5.40
PHP Version: 7.1.13-nts

Description
I did this for some properties of the model
class User extends Model
{
    protected $casts = [
        'status' => 'array'
    ];
}

I setstatusto array type, and if my database is inserted, the defaultstatusis null or null will also return a null to me. That's not what I wanted. I think that when I setstatusto array type, even if the value in my database is null or null.laravel should return an empty array for me, this is the correct type.

Just use accessor.
return $value ?? [];

I get what you are raising tho.

I'm not asking how to fix the problem, it's that the problem is actually. PHP is a weak type, but I set the property to an array, and I should force an array type to return, regardless of his data.


Add a way to intialize collectors for specific tagmaps

Normally when a metric is created, it will not report anything until values are added to it.  We can't simply start reporting 0 since we don't know what tag key/values will be used, but in many cases we use metrics without any tags or with a known set of fixed tags.  In those cases it would be nice to be able to force collectors to start immediately reporting 0, so we don't have gaps in the resulting time-series data when services are restarted.
I see two possible ways to address this:

Give every collector some kind of "touch" or "reset" function that accepts a TagMap.  This would essentially create a 0 value for the given tagmap.
Add an initialZeros configuration setting that accepts a sequence of TagMaps and basically does the 1. itself.

Or we could do both.

I'd vote to close this until we have a better use case. If we have a metric that tends to be zero, then we use a counter and set it to zero on service startup. It is occasionally annoying when a new service doesn't report /errors, but it usually resolves itself :)
cc @aliyakamercan


Memory usage estimator

GPU memory is limited. It would be great to have an utility function to estimate the memory usage of training a model given the shape of the input. Currently, the only way is to run the model with different hidden_size and batch_size (trial and error). MXNet could provide an API that makes this process automatic so that it's easier for the user. @Roshrini @pinaraws  @cgraywang @yifeim feel free to add additional contexts
For example, there is a 3rdparty library that prints the theoretical memory consumption of a pytorch model's forward/backward intermediate data entries, weights and gradients. https://github.com/jacobkimmel/pytorch_modelsize
In MXNet we can record the memory planning of a model and report the memory usage given input shapes. This does not include the temporary memory requested at runtime (e.g. by MKLDNN/CUDNN). If reporting planned memory usage not accurate enough, we can simply run the trials and return the actual peak memory usage at runtime.

Curious to know why we need an estimator. Can a estimated number stop the users from trying a larger input size? I think they would not stop until they see the OOM error.
Compared with memory usage estimator, I think it would be better if MXNet can have a memory profiler.

@TaoLv This feature is more geared towards federated training, when some of the model shapes are inferred from the user-provided dataset. In this case, trying to utilize memory more efficiently, we have to understand how to set other parameters, e.g., batch_size, accordingly. Essentially, we aim to "try a larger input size" inside a script.
An example would be to fit word_language_model with different vocabulary sizes, where filtering by word frequency is not an option.
Additionally, it would be useful to consider memory planning for sparse data. In this case, we would provide the 'max_num_numzeros' parameter to reduce the uncertainties.


[MAF] Add function to create rainfall plots

https://www.biostars.org/p/9477950/#9478012

Implemented in the 0.17.0-dev branch.


make HTTP routes async

this doesn't change any logic, but as far as I understand sync routes use threads instead of the starlette event loop: https://fastapi.tiangolo.com/async/#path-operation-functions

Do we have any specific examples of someone actually using that, or is hypothetical?
I think it will be required somewhere down the line, I put it up front here because it can be reviewed as an independent PR

Yes, in that if I remember correctly, we broke it previously while working on Cog and had to put it back.

we could stick a timeout or something where the endpoint waits for the worker for a similar time as what the tcp timeout would have been? do you remember what exactly the use case was and why they didn't want to use retries instead?


Increase test coverage for regularizers_test

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (aa02010) 78.89% compared to head (715ff0c) 56.26%.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #18727       +/-   ##
===========================================
- Coverage   78.89%   56.26%   -22.63%
===========================================
  Files         336      336
  Lines       33934    33934
  Branches     6651     6651
===========================================
- Hits        26771    19094     -7677
- Misses       5584    13393     +7809
+ Partials     1579     1447      -132

Flag
Coverage Δ

keras
56.26% <ø> (-22.52%)
:arrow_down:

keras-jax
?

keras-numpy
56.26% <ø> (+0.01%)
:arrow_up:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 184 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Aliases with period separator are disregarded

Version - 2.1.10
Having an issue where the aliases are being removed if using a period separator. For any other separator, the aliases stay intact.
Example:
test.id (test.id)
will output
id
test.id (test_id)
will output
test_id
private const DB_TABLE = 'test';
private const COLUMN_REF = 'test.';

$columns = [
	self::DB_TABLE . '.id (' . self::COLUMN_REF . 'id)',
	self::DB_TABLE . '.join_id (' . self::COLUMN_REF . 'join_id)',
	self::DB_TABLE . '.amount (' . self::COLUMN_REF . 'amount)',
	self::DB_TABLE . '.title (' . self::COLUMN_REF . 'title)',
	self::DB_TABLE . '.status (' . self::COLUMN_REF . 'status)',
	self::DB_TABLE . '.username (' . self::COLUMN_REF . 'username)',
	self::DB_TABLE . '.' . self::CREATED_COLUMN . ' (' . self::COLUMN_REF . self::CREATED_COLUMN . ')',
	self::DB_TABLE . '.' . self::CREATED_BY_COLUMN . ' (' . self::COLUMN_REF . self::CREATED_BY_COLUMN . ')',
	self::DB_TABLE . '.' . self::DELETED_COLUMN . ' (' . self::COLUMN_REF . self::DELETED_COLUMN . ')',
	self::DB_TABLE . '.' . self::DELETED_BY_COLUMN . ' (' . self::COLUMN_REF . self::DELETED_BY_COLUMN . ')',
	self::DB_TABLE . '.' . self::UPDATED_COLUMN . ' (' . self::COLUMN_REF . self::UPDATED_COLUMN . ')',
	self::DB_TABLE . '.' . self::UPDATED_BY_COLUMN . ' (' . self::COLUMN_REF . self::UPDATED_BY_COLUMN . ')',
];

return $this->dbInit->select(
	self::DB_TABLE,
	$columns
	BaseModel::getConditions()
);

Output QUERY:
SELECT test.id,test.join_id,test.amount,test.title,test.status,test.username,test.created,test.createdBy,test.deleted,test.deletedBy,test.updated,test.updatedByFROMtestORDER BYtest.id DESC LIMIT 10 OFFSET 0
Expected Output:
SELECT
`test`.`id` as `test.id`,
`test`.`join_id` as `test.join_id`,
`test`.`amount` as `test.amount`,
`test`.`title` as `test.title`,
`test`.`status` as `test.status`,
`test`.`username` as `test.username`,
`test`.`created` as `test.created`,
`test`.`createdBy` as `test.createdBy`,
`test`.`deleted` as `test.deleted`,
`test`.`deletedBy` as `test.deletedBy`,
`test`.`updated` as `test.updated`,
`test`.`updatedBy` as `test.updatedBy`
FROM `test` ORDER BY `test`.`id` DESC LIMIT 10 OFFSET 0

In SQL, the alias character is accepted A-Z, 0-9, and _, other characters are not recommended.
And there's no need to use the alias for your case. You can refer to the column directly with test.id, or replace the alias name to test_id or testId.

@webmdt Thanks for the sample. I need to test more databases for this case and check out what alias characters can be accepted.
However, to avoid the name conflict, the best practice is to adopt different naming styles for the table and the column, or any way that can return different strings. It's up to the developer because as a database framework, it cannot check the name conflict problem.
Also, if you want to allow the alias to accept the period ., you can replace <alias>[\p{L}_][\p{L}\p{N}@$#\-_] to<EMAIL_ADDRESS>(only four places in the project), and test whether it can resolve your problem just now.


[bug]: Nullable causes model binding error in webhooks

Describe the bug
Enabling nullability in my operator project causes webhooks to fail.  When nullability is enabled, I see the following Request.Object.Uid is required error on the kube-apiserver, which it is receiving from ASP.NET:
I0425 03:24:52.491708       1 request.go:1411] body was not decodable (unable to check for Status): Object 'Kind' is missing in '{"type":"https://tools.ietf.org/html/rfc9110#section-15.5.1","title":"One or more validation errors occurred.","status":400,"errors":{"Request.Object.Uid":["The Uid field is required."]},"traceId":"00-1e0d8715878c6606b3c70f8683bd85d3-c57cbcf6dfc4b1a6-00"}' W0425 03:24:52.491764       1 dispatcher.go:225] Failed calling webhook, failing closed mutate.test.company.com.v1: failed calling webhook "mutate.test.company.com.v1": failed to call webhook: the server rejected our request for an unknown reason
And the AdmissionReview Kubernetes is sending in the POST has the following fields for the object (metadata and spec removed for brevity):
{
  "kind": "AdmissionReview",
  "apiVersion": "admission.k8s.io/v1",
  "request": {
    "object": {
      "apiVersion": "company.com/v1",
      "kind": "test",
      "metadata": "my metadata"
      "spec": "my spec"
    },
  }
}

And what little MVC logging there was:
trce: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[52]       Action Filter: Before executing OnActionExecuting on filter Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter. dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ModelStateInvalidFilter[1]       The request has model state errors, returning an error response.
It seems that the model binder is trying to bind the uid field for the object, but because 1) nullable is enabled, 2) the Uid is not nullable in k8s.models.V1ObjectMeta, and 3) the request does not contain the uid under the request.object, it is failing to bind.
Some Microsoft documentation might be helpful.
To reproduce

Run Kubernetes API server with verbose logging.
Create a new operator project with a mutation webhook and nullable enabled.
Implement the webhook .
Run the operator.
Try to create the object in Kubernetes.
Observe that the Kubernetes API server logs the error given by ASP.NET.
Disable nullable in the project.
Do steps 4-5 the and observe that the request finishes (though mine is still failing after the 200 status for unknown reasons).

Expected behavior
Model binding should succeed with nullable-enabled projects.
Screenshots
No response
Additional Context
No response

I meant to add to this that I used that Microsoft doc to add this line:
 builder.Services
     .AddControllers(o => o.SuppressImplicitRequiredAttributeForNonNullableReferenceTypes = true);

That seems to have fixed the issue. However, I think there is a chance that anyone using webhooks could run into this, and it was a pretty obscure problem to diagnose. I realize this should probably be fixed in the k8s lib, but it would be nice if the operator had a workaround until kubernetes-client/csharp#606 is fixed.

Hey @ian-buse
Is this still an issue? as far as I saw, they closed the issue over at the k8s client.

They closed it because they don't have enough contributors to fix it. :( It's still a problem.

Hey @ian-buse
Just encountered this specific issue while updating the dependencies in #764. When I set all these fields as "required" (via [Required] attribute), then the linter is pleased. However this does not seem to fix the error.
If I understand you correctly, the request that is sent does not match the nullable enabled model?
Do you have any idea how we might fix this issue? I don't know if I'm able to allow a lax model binding per request.
If

The only thing I've found that fixed the issue is using Action<MvcOptions> when adding the controllers:
var builder = WebApplication.CreateBuilder(args);
 builder.Services
     .AddControllers(o => o.SuppressImplicitRequiredAttributeForNonNullableReferenceTypes = true);

Yeah, the k8s library doesn't support nullable types, and in MVC, the reference types (like the UID string) are considered required by default if they are not nullable. So, when MVC tries to bind the UID and finds it missing in the webhook, it fails because the model expects it to be there.
I looked into adding something in KubeOps.Operator.ServiceCollectionExtensions.AddKubernetesOperator(), but it would require adding a dependency on AspNetCore to the core operator. Maybe an extension method in Operator.Web would work better instead? This was just a quick writeup, haven't tested it:
using KubeOps.Abstractions.Builder;
using KubeOps.Operator.Builder;

using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.DependencyInjection;

namespace KubeOps.Operator.Web;

/// <summary>
/// Method extensions for the <see cref="IServiceCollection"/>.
/// </summary>
public static class ServiceCollectionExtensions
{
    /// <summary>
    /// Add the Kubernetes operator to the dependency injection.
    /// </summary>
    /// <param name="services"><see cref="IServiceCollection"/>.</param>
    /// <param name="operatorConfigure">An optional configure action for adjusting settings in the operator.</param>
    /// <param name="mvcConfigure">An optional configure action for the webhook MVCs. Defaults to suppressing required
    /// attributes for non-nullable reference types if no configuration is provided.</param>
    /// <returns>An <see cref="IOperatorBuilder"/> for further configuration and chaining.</returns>
    public static IOperatorBuilder AddKubernetesOperator(
        this IServiceCollection services,
        Action<OperatorSettings>? operatorConfigure = null,
        Action<MvcOptions>? mvcConfigure = null)
    {
        services
            .AddControllers(
                mvcConfigure ?? (o => o.SuppressImplicitRequiredAttributeForNonNullableReferenceTypes = true));

        var settings = new OperatorSettings();
        operatorConfigure?.Invoke(settings);
        return new OperatorBuilder(services, settings);
    }
}

This would require making the OperatorBuilder public.
Or, maybe some documentation on the issue would suffice, like adding it to the examples and the README.


Error: Node Sass does not yet support your current environment: Windows 64-bit with Unsupported runtime (93)

Target SharePoint environment
SharePoint Online
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Developer environment
Windows
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[X] 💥 Microsoft Edge
[ ] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

Describe the bug / error
Error:
Error: Node Sass does not yet support your current environment: Windows 64-bit with Unsupported runtime (93)
For more information on which environments are supported please see:
https://github.com/sass/node-sass/releases/tag/v4.14.1
at module.exports (C:\SPFx\LAMetroPCC\node_modules\node-sass\lib\binding.js:13:13)
at Object. (C:\SPFx\LAMetroPCC\node_modules\node-sass\lib\index.js:14:35)
at Module._compile (node:internal/modules/cjs/loader:1126:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
at Module.load (node:internal/modules/cjs/loader:1004:32)
at Function.Module._load (node:internal/modules/cjs/loader:839:12)
at Module.require (node:internal/modules/cjs/loader:1028:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (C:\SPFx\LAMetroPCC\node_modules@microsoft\gulp-core-build-sass\lib\SassTask.js:12:18)
at Module._compile (node:internal/modules/cjs/loader:1126:14)
Steps to reproduce

Installed Node.js version 16.17.1

Expected behavior
Thought I could use latest version of node js

What does the package.json from your project look like? It seems like you have SPFx 1.12.1 installed.

"dependencies": {
"@fluentui/react": "^8.22.2",
"@microsoft/sp-core-library": "1.12.1",
"@microsoft/sp-lodash-subset": "1.12.1",
"@microsoft/sp-office-ui-fabric-core": "1.12.1",
"@microsoft/sp-property-pane": "1.12.1",
"@microsoft/sp-webpart-base": "1.12.1",
"@pnp/graph": "^2.15.0",
"@pnp/sp": "^2.15.0",
"office-ui-fabric-react": "7.156.0",
"react": "16.9.0",
"react-dom": "16.9.0",
"react-excel-renderer": "^1.1.0",
"sass": "^1.58.3",
"semantic-ui-react": "^1.3.1"
},
"devDependencies": {
<EMAIL_ADDRESS>"0.2.3",
"@microsoft/sp-build-web": "1.12.1",
"@microsoft/sp-module-interfaces": "1.12.1",
"@types/react": "16.9.36",
"@types/react-dom": "16.9.8",
"@types/webpack-env": "1.13.1",
"ajv": "~5.2.2",
"gulp": "~4.0.2"
}

@pmeripe SPFx 1.12.1 doesn't support node v16.
Please, take a look at the compatibility table.
You should use node v10, v12, or v14.

Hi Alex,
Thanks for your reply. So If I want to upgrade all to latest, what are the steps?

I would recommend using CLI for Microsoft 365 to upgrade the project correctly to the latest version.

Thanks Alex. I downgraded Node.js to 14.21.3 and tried to fix other errors. But still I get webpart error.
Please help!


Some bug using Jupyter console

jupyter console --kernel=YOUR_KERNEL is a great way to quickly test a freshly compiled kermel but I have noticed some bugs using it.

The Out[1]: line is always displayed even when the returned result is null. See https://github.com/hadim/scijava-jupyter-kernel/issues/23

After the first line has been executed, the other lines shows multiples prompts to the user. See https://github.com/hadim/scijava-jupyter-kernel/issues/18

Hi thanks for the bug report.

I think null should be displayed.  we have another value OutputCell.HIDDEN that gets rid of the Out.  there are a bunch of runtime functions that should return this and so I agree there are too many circumstances where you see Out.  see https://github.com/twosigma/beakerx/issues/5102

I will look into this thank you very much.

Thanks @scottdraves

1 is done https://github.com/twosigma/beakerx/pull/5199

I still have the issue with my kernel. Here is the code that set the result object to SimpleEvaluationObject : https://github.com/hadim/scijava-jupyter-kernel/blob/master/src/main/java/org/scijava/jupyter/kernel/evaluator/Worker.java#L102
I don't think I am doing anything wrong because the equivalent code in your Groovy kernel is still the same.

It should be fixed with https://github.com/twosigma/beakerx/pull/5205

I am seeing this:
$ jupyter console --kernel=Groovy
[main] INFO com.twosigma.jupyter.KernelConfigurationFile - Path to config file : /Users/spot/Library/Jupyter/runtime/kernel-17278.json
[Thread-0] INFO com.twosigma.beaker.groovy.handler.GroovyKernelInfoHandler - Processing kernel info request
[Thread-0] INFO com.twosigma.beaker.groovy.handler.GroovyKernelInfoHandler - Processing kernel info request
/Users/spot/anaconda/envs/beakerx/lib/python3.5/site-packages/jupyter_console/ptshell.py:94: UserWarning: No lexer found for language ''. Treating as plain text.
  warn("No lexer found for language %r. Treating as plain text." % name)
Jupyter console 5.1.0

BeakerX kernel for Apache Groovy

In [1]: 1+2/Users/spot/anaconda/envs/beakerx/lib/python3.5/site-packages/jupyter_console/ptshell.py:599: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kern.
  warn('The kernel did not respond to an is_complete_request. '
In [1]: 1+2
[groovy worker] ERROR com.twosigma.beaker.jupyter.msg.MessageCreator - Unhandled status of SimpleEvaluationObject : RUNNING
Out[1]: 3

In [2]: 2+3
      :
      :
[groovy worker] ERROR com.twosigma.beaker.jupyter.msg.MessageCreator - Unhandled status of SimpleEvaluationObject : RUNNING
Out[2]: 5

There are a couple of problems:

excessive info logging "Processing kernel info request" I just fixed this.
"no lexer found".  this is a bug, will fix: https://github.com/twosigma/beakerx/issues/5206
the multiple prompt problem.  i will have to look at 5205 and see how to get its fix without changing the null behavior.  thank you very much for the PR.

I don't have error 1 and 2 (I guess it's because the scijava kernel is doing thing little differently).
Good for 3.
Also there a Python UserWarning when the first cell is executed : UserWarning: The kernel did not respond to an is_complete_request. Setting use_kern.warn('The kernel did not respond to an is_complete_request. '`

https://github.com/twosigma/beakerx/issues/5251 for the is_complete_request

these are all fixed now, @hadim do you agree this can be closed?

$ jupyter console --kernel=Groovy
Jupyter console 5.1.0

BeakerX kernel for Apache Groovy

In [1]: 1+2
Out[1]: 3

In [2]: 3+4
Out[2]: 7

In [3]:
Do you really want to exit ([y]/n)? y
Shutting down kernel

Good for me !


Code formatting on working on GitHub

This may be due to the GitHub redesign, but I'm not able to get it working at all with what's currently in the repo. Will try to look into it and publish a working version ASAP.

Hmm it's working now! Sorry for the noise.


Fix the production build tasks chaining

See https://github.com/dsebastien/modernWebDevGenerator/issues/66 for details

I see the same issue. Will this be fixed soon?

You can try it again, I think it should be fixed now.

This still does not work.
[19:06:07] Error in plugin 'run-sequence(scripts-javascript-dist)'
Message:
    Error on fetch for app.js at file:///D:/Program/my-new-project/app.js
        Loading .tmp\core\boot.js
        ENOENT: no such file or directory, open 'D:\Program\my-new-project\app.js'
Details:
    originalErr: Error: ENOENT: no such file or directory, open 'D:\Program\my-new-project\app.js'


htmlq: 0.2.0 -> 0.3.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/mgdm/htmlq/releases.
meta.description for htmlq is: "Like jq, but for HTML"
meta.homepage for htmlq is: "https://github.com/mgdm/htmlq"
meta.changelog for htmlq is: ""
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

/nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0/bin/htmlq passed the binary check.

1 of 1 passed binary check by having a zero exit code.

1 of 1 passed binary check by having the new version present in output.

found 0.3.0 with grep in /nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0

found 0.3.0 in filename of file in /nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0

directory tree listing: https://gist.github.com/3f341e991f967bb140b1a6918dcaa04b

du listing: https://gist.github.com/ab46b96ba13923707f2b019ca8a18bcf

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
htmlq

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A htmlq https://github.com/r-ryantm/nixpkgs/archive/4bcadfca0463e86d38be1e28d9abddd9bcefe71a.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0
ls -la /nix/store/5pz8ynz4z76r1z2cnwihwd7yqk3mv8hv-htmlq-0.3.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

htmlq

Maintainer pings
cc @siraben @nerdypepper for testing.

Result of nixpkgs-review pr 138122 run on x86_64-linux 1

1 package built:

htmlq


Control how the CloudFront cache is invalidated

Start from the Use-case
I am running a single page react app. I have 10 different JS files (chunks) after I build my app.
It is normal that a user request 3-5 js files while they are using a part of the application. When they browse to another part they will download more files.
There is an issue if I decide to deploy a new version. Then the additional files are removed from the S3 and CDN. Now when the user asks for the additional files they get a 404.
I suggest to allow us to keep some files on S3 or CloudFront.
Of course I would be open for alternative solutions
Example Config
No response
Implementation Idea
No response

I wonder if we could (and if S3 allows it), instead of deleting "obsolete" files, set them to expire (be deleted) after 1 day 🤔 I haven't researched that at all but that would be a solution that we could apply by default for everyone?

Probably related to https://github.com/getlift/lift/issues/398

@mnapoli yep, I was researching on the very same right now. I wonder if we could tag all obsolete files and then use a lifecycle rule to delete these tagged files after a day. The question is whether there is an efficient way to compare and add those tags.

Why not, as long as it works


Extract Exif Geocode Data on creating observation

The add image javascript on multiple image uploader is now smart enough to detect GPS coordinates and altitude and give the user the option to apply these coordinates to the observation they are creating.

What the heck. Something is wrong here.  I'll look at it. I really wish I could run these tests on my windows machine.

Give Ray's latest comment and the branch conflicts should I close this?

Closing.  Just saw the comment about #178 including this code as well.


Added my respitory and alphabetical order

Bam, resolved @sanket-deepsource

Bam, resolved @sanket-deepsource

Can anyone review this?

Can anyone review this?


groups: Enable group-settings value on groups in production.

The main change is redefining ALLOW_GROUP_VALUED_SETTINGS to not control code, but instead to instead control the configuration for whether settings that have not been converted to use our modern UI patterns should require system groups.
Fundamentally, it's the same for the realm/stream group-valued settings, which don't have the new UI patterns yet.
We remove the visual hiding of the "can manage group" setting, which was hidden only due to transitions being incomplete.

Merged, since doing so seems helpful for keeping the queue of API changes moving.

Looks good.


debug watch duration

Skipping the format changes, I found that watchDuration is updated by currentLocation. The real problem is that the items in watchedDataPoints is always {start: 0, end: 0}, so aggWatchedDuration is always zero. You can debug from here.

@qiushiyan Actually watchedDataPoints (and most of the other variables for the fact) will update only when watchDuration updates. And watchDuration in the handleProgress function is not updating. Pasting handleProgress here your reference.
Maybe I am wrong here. Can you please elaborate on what the issue is. Thanks!
const handleProgress = (
		currentLocation: number,
		currentTimestamp: number,
	) => {
		// console.log("handleprogress - watchDuration", watchDuration);
		updateTabSwitch(currentLocation, currentTimestamp);
		updateOutOfFocus(currentLocation, currentTimestamp);
		updateSkippedAndWatchedLocations(currentLocation, currentTimestamp);
		setWatchDuration(currentLocation);
}

Yeah, it's updating. So the issue is not what I was thinking it to be. Let me take a deeper look into it. Thanks for your help @qiushiyan

@qiushiyan Quick question -  I can see watchDuration updating in return statement but not in any other function. For example, watchDuration remains 0 in updateSkippedAndWatchedLocations but keeps on updating on the screen (from the return statement).

@qiushiyan Another thing, watchDuration is correctly updating in handlePlaybackRateChange.  I am suspecting this is because handleProgress is being called from useEffect which is dependent on player which is initialized only once when youtube is ready to play the video - and here the watchDuration is 0.


RFE: Patch Installer Manifests

Add an enhancement doc proposing API changes to allow hive consumers to specify arbitrary edits to the manifests generated by openshift-install create manifests prior to feeding them to openshift-install create cluster.
HIVE-1793

/cc @abraverm
Thought you might be interested to see how ClusterDeploymentCustomization is evolving :)

Finished this out alongside the code (#2499). Ready for review.


Kafka 2257

This is a 'fake' pull request created solely for the purpose of getting an initial review on this branch before I do the official pull request against the apache/kafka repo.
Hence, this will not be merged into the KAFKA-2276 branch.
@ewencp @junrao
This patch is intended to replace the existing replication test suite, and follows the proposal for replication test suite outlined here: https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests

@granders This patch is smaller than I thought it was going to be and much of it is for utilities that can be reused in other tests -- this is a great sign that we're on the right track! I left a bunch of comments but haven't actually tested yet, so take everything with a grain of salt :)

Done with this local PR


binary_to_hex / hex_to_binary

We could start with scalar implementations and work our way up. Importantly, we need reasonable benchmarks.


about face profiling

could you have python version about face profiling?

Nope for now.

I have made a python version

Looking forward to your Python version. @zengxianyu

Looking forward to your Python version. @zengxianyu

https://github.com/zengxianyu/facepf

Looking forward to your Python version. @zengxianyu

https://github.com/zengxianyu/facepf

It seems that many files in the code are missing. A clear readme may help users : )

Looking forward to your Python version. @zengxianyu

https://github.com/zengxianyu/facepf

It seems that many files in the code are missing. A clear readme may help users : )

I'll complete the readme later

I have completed the readme and provided an example

@zengxianyu Does your python code can rotate face for BFM model?

@zengxianyu Does your python code can rotate face for BFM model?

I'll add support for bfm later

@zengxianyu so for each new face will have to generate the .npy right?

@zengxianyu so for each new face will have to generate the .npy right?

Yes

@cleardusk @zengxianyu

I have made a python version

is it possible to generate face data with extreme expression?


flutter slow on recent PowerVR GPUs

Hello,
my company is currently working together with Texas Instruments to make flutter run on their AM62xx boards. We found that the performance is not that good, and as it turns out it's worse than it could be, because skia enables (apparently) unnecessary GL workarounds when it finds a PowerVR GPU.
I also filed a skia issue for this but I thought having it as a flutter issue might be good as well, since it primarily manifests itself to the user as "flutter rendering slowly on that platform". Lmk if you think the skia issue is enough.
Details

Use a board with a recent PowerVR GPU (like the Texas Instruments SK-AM62)
Build a system for it using yocto, meta-flutter and my flutter-pi patches
Run the included flutter gallery app on the board
Discover that FPS are pretty bad (maybe 10FPS), even though the GPU is not that far away from the VideoCore GPU in a Pi 4.
Profile & find out that most of the CPU time is spent inside some functions like SoftwarePathRenderer::draw, with drawTextBlob a bit up the call stack.
Turns out skia enables lots of workarounds on PowerVR Rogue GPUs: https://github.com/google/skia/blob/8941e05187e88a97d1db8434d5ec484304b66081/src/gpu/ganesh/gl/GrGLCaps.cpp#L4266-L4275 (just search for kPowerVRRogue in that file)
If you disable all the workarounds, it still renders perfectly fine but now at 50-60FPS.

I think those workarounds are primarily in place for older PowerVR GPUs, some (maybe even all) of them might be unnecessary on newer GPUs & drivers, like the AXE-1-16M that's present on the SK-AM62.

Hi @ardera, which Flutter version are you facing this issue?

@huycozy This is with latest flutter stable 3.3.4, but all the skia workarounds are still there in latest skia, so you'll also face this in flutter master

@ardera I'm not sure I can go deeper with this since I'm not familiar with embedded devices. I will label this issue for further investigation from the team. Thanks!

@ardera I'm not sure I can go deeper with this since I'm not familiar with embedded devices. I will label this issue for further investigation from the team. Thanks!

Yep sure, I mean it's impossible to reproduce this anyway without a PowerVR GPU.

@egdaniel @bsalomon Is there any way to make the PowerVR carveout finer-grained?

Absolutely! What we'd need is the GL_VERSION and GL_RENDERER strings from a newer device that works OK and then we can add detection for a newer driver version.

@bsalomon
This is the output of kmscube on the SK-AM62xx (dunfell):
root@am62xx-evm:~# kmscube -D /dev/dri/card0
Using display 0x31023d90 with EGL version 1.5
===================================
EGL information:
  version: "1.5"
  vendor: "Mesa Project"
  client extensions: "EGL_EXT_client_extensions EGL_EXT_device_base EGL_EXT_device_enumeration EGL_EXT_device_query EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses EGL_KHR_debug EGL_EXT_platform_device EGL_EXT_platform_wayland EGL_KHR_platform_wayland EGL_MESA_platform_gbm EGL_KHR_platform_gbm EGL_MESA_platform_surfaceless"
  display extensions: "EGL_ANDROID_native_fence_sync EGL_EXT_buffer_age EGL_EXT_create_context_robustness EGL_EXT_image_dma_buf_import EGL_EXT_image_dma_buf_import_modifiers EGL_EXT_image_gl_colorspace EGL_EXT_yuv_surface EGL_IMG_context_priority EGL_KHR_cl_event2 EGL_KHR_config_attribs EGL_KHR_create_context EGL_KHR_fence_sync EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_image EGL_KHR_image_base EGL_KHR_image_pixmap EGL_KHR_no_config_context EGL_KHR_partial_update EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_EXT_pixel_format_float EGL_KHR_wait_sync EGL_MESA_configless_context EGL_MESA_drm_image EGL_MESA_image_dma_buf_export EGL_WL_bind_wayland_display EGL_IMG_cl_image "
===================================
OpenGL ES 2.x information:
  version: "OpenGL ES 3.1 build 1.15@6133109"
  shading language version: "OpenGL ES GLSL ES 3.10 build 1.15@6133109"
  vendor: "Imagination Technologies"
  renderer: "PowerVR Rogue AXE-1-16M"
  extensions: "GL_APPLE_texture_format_BGRA8888 GL_EXT_blend_minmax GL_EXT_buffer_storage GL_EXT_clip_control GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_compressed_ETC1_RGB8_sub_texture GL_EXT_conservative_depth GL_EXT_copy_image GL_EXT_discard_framebuffer GL_EXT_draw_buffers GL_EXT_draw_buffers_indexed GL_EXT_draw_elements_base_vertex GL_EXT_EGL_image_array GL_EXT_float_blend GL_EXT_gpu_shader5 GL_EXT_memory_object GL_EXT_memory_object_fd GL_EXT_multi_draw_arrays GL_EXT_multisampled_render_to_texture GL_EXT_multisampled_render_to_texture2 GL_EXT_occlusion_query_boolean GL_EXT_polygon_offset_clamp GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_separate_shader_objects GL_EXT_shader_framebuffer_fetch GL_EXT_shader_group_vote GL_EXT_shader_implicit_conversions GL_EXT_shader_io_blocks GL_EXT_shader_non_constant_global_initializers GL_EXT_shader_pixel_local_storage GL_EXT_shader_pixel_local_storage2 GL_EXT_shader_texture_lod GL_EXT_shadow_samplers GL_EXT_sparse_texture GL_EXT_sRGB_write_control GL_EXT_texture_border_clamp GL_EXT_texture_buffer GL_EXT_texture_cube_map_array GL_EXT_texture_format_BGRA8888 GL_EXT_texture_format_sRGB_override GL_EXT_texture_rg GL_EXT_texture_shadow_lod GL_EXT_texture_sRGB_decode GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_RG8 GL_EXT_YUV_target GL_IMG_framebuffer_downsample GL_IMG_multisampled_render_to_texture GL_IMG_program_binary GL_IMG_read_format GL_IMG_shader_binary GL_IMG_texture_format_BGRA8888 GL_IMG_texture_npot GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_KHR_debug GL_KHR_robustness GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth24 GL_OES_depth_texture GL_OES_draw_buffers_indexed GL_OES_draw_elements_base_vertex GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_EGL_sync GL_OES_element_index_uint GL_OES_fragment_precision_high GL_OES_get_program_binary GL_OES_gpu_shader5 GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_required_internalformat GL_OES_rgb8_rgba8 GL_OES_sample_shading GL_OES_sample_variables GL_OES_shader_image_atomic GL_OES_shader_io_blocks GL_OES_shader_multisample_interpolation GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_border_clamp GL_OES_texture_buffer GL_OES_texture_cube_map_array GL_OES_texture_float GL_OES_texture_half_float GL_OES_texture_npot GL_OES_texture_stencil8 GL_OES_texture_storage_multisample_2d_array GL_OES_vertex_array_object GL_OES_vertex_half_float"
===================================

I can also test on kirkstone if you want, the driver may be different there.

@bsalomon says https://skia-review.googlesource.com/c/skia/+/596218 will fix this.

That change has now landed and rolled to Flutter Engine.

@ardera Please check this out on master and let us know if it's fixed! Thanks for the report!

Okay, just tested, I can confirm it's a lot better. Thanks, I appreciate the fast fix!
Though not 100% sure if it's the same as having all the other workarounds disabled too.
Also, when looking at it in PVRCarbon, I got lots of performance warnings about render target switches. The skia "reduce ops task splitting feature"^1 helped reduce that, but it's disabled in the engine^2 (because it increases peak graphics memory usage, AFAIK).
Should I provide some more concrete performance numbers (I'm not even sure how much that improved things) and submit issues for that too?

I can't speak for the reduce-ops-task decision, but if other workarounds don't seem to be required and disabling them improves performance I'd be interested in disabling them in Skia.

I'm closing this issue as it was tracking removal of features being disabled on PVR GPUs. If there are other improvement backend by use cases to be made, lets file separate issues for clarity.


Keep the Modal open until user closes

NOTE: this issue is also resolving the problem that the background is not interactive whenever the modal is active.


Create a release Github action

A release Github action should be created, so that it is possible to automatically generate the release when a push to a particular tag is detected.
An example:
name: Main

on:
  push:
    tags:
      - "v*.*.*"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Release
        uses: softprops/action-gh-release@v2

In this case, the release will be automatically created. Then, it can be used to include a badge in README.md with latest release, which helps for its visualization

Won't be implemented due to migration to OpenShift organization and usage of Konflux CI/CD


Add more security related attributes to webview

Allowing images but not scripts is a significant security improvement when embedding apps in webview elements.
There is apparently a code-level possibility of controlling this, but being exposed as a set of prefs, it currently applies only to BrowserWindow and is not exposed as an attribute of the webview element at all.
Proposals:

make the disablewebsecurity able to handle non-truth values such as script and display

or

add new tag(s) to selectively allow display and/or run of insecure resources

Dependency on https://github.com/lloeki/matterfront/pull/41

I'm merging this to #2749.


Request to add the feature to get the exact image array size back after applying register.

Hi,
Please let me know if there is any straightforward way to get the full 4096 by 4096 array back after applying the register function. I checked that aia_prep.pro in SSWIDL is not doing this resizing. help needed.

Hello @dinesh-hegde,
This is known behavior with that function that we had to add a warning note to the function.
The affine transform has to cut off pixels to account for the de-rotation correctly.
If you do need to go back to the original size of 4096 by 4096, the current best method is to do something like:
if tempmap.data.shape == (4094, 4094):
	padded_array = np.pad(tempmap.data, 1, mode="constant")
	tempmap.meta["crpix1"] += 1
	tempmap.meta["crpix2"] += 1
	newmap = tempmap._new_instance(padded_array, tempmap.meta)

What was the output array size?
Do you need it to be 4096 by 4096?
Are you planning to run the data through a DEM code?

Hi @nabobalis,
Thank you. Do you think the resized output size is always 4094 by 4094? After looking at the warning on the documentation page,I was just thinking whether this manual padding of some constant value is ok or not.
Output array was 4094 by 4094.
Yes, I need it to be 4096 by 4096.
No, working on some ML project.


Installation error "WinError 193" when attempting install.bat from ComfyUI\custom_nodes\was-node-suite-comfyui

When I attempt to run the included install.bat in the "was-node" folder, I receive the following error:

I receive this error with all three installation methods seen here:

I have ensured that I have admin properties and those folders have write permissions, just to be sure, and still nothing. I've tried troubleshooting using fixes found elsewhere but none of them have worked such as: removing * from the numpy requirments, manually installing torch, uninstalling and reinstalling numpy manually, etc. I have of course tried restarting the pc, same error. I don't know what else to try.

After a bit of work I can say for certain it's the "git+https://github.com/WASasquatch/img2texture.git" and "git+https://github.com/WASasquatch/img2texture.git" commands, taking them out lets it load fine. That has to mean I have an issue with my git right? But I tried a reinstall, and it's definitely in my PATH:

I attempted to manually do the git clones in the folder shown, was that the right place to do that?

Even after this though they don't load in comfy

I was able to resolve this issue on my own. I tried changing the location of my GIT location and moving the path higher up in my PATH, and running it again. I don't know ultimately what fixed the issue but I got everything to install for now at least and can finally use the templates people have out there that all seem to include this lol.


[iOS] Error: could not build on iOS after link

Versions
react v16.1.3
react-native v0.55.2
react-native-google-signin: 0.12.0
xcode v9
Issue
I added Google Signin sdk and it build success, but has error can not read BUTTON_SIZE_ICON of undefined. After I link react-native-google-signin, it can't build with error
...
fatal error: could not build module 'CoreFoundation'
#include <CoreFoundation/CoreFoundation.h>
 ~~~~~~~~^
...
warning: non-portable path to file '"RNGoogleSignin.h"'; specified path differs in case from file name on disk [-Wnonportable-include-path]
#import "RNGoogleSignIn.h"
        ^~~~~~~~~~~~~~~~~~
        "RNGoogleSignin.h"
...
fatal error: could not build module 'Foundation'
#include <Foundation/Foundation.h>
 ~~~~~~~~^
1 error generated.
3 errors generated.

** BUILD FAILED **

The following commands produced analyzer issues:
        Analyze Base/RCTModuleMethod.mm normal x86_64
(1 command with analyzer issues)
...
No such file or directory
Print: Entry, ":CFBundleIdentifier", Does Not Exist

Command failed: /usr/libexec/PlistBuddy -c Print:CFBundleIdentifier build/Build/Products/Debug-iphonesimulator/demo.app/Info.plist
Print: Entry, ":CFBundleIdentifier", Does Not Exist
...

I tried a lots of way but it not working.

I follow https://github.com/devfd/react-native-google-signin/issues/338#issuecomment-380648271 and fixed this issue, but have another way?

Another common build issue closed with the only advice "try example app".  Not very helpful.


Dispatch sender display

Describe the feature
Officer -> Dashboard
A display that appears under: "Active 911 Calls" represents whether an assignment (callout) is from a caller directly or via Dispatch.
Additional Context
No response

So that one has there still another point which is called e.g.: "From" and there either "Dispatch" or "Caller" stands.
So you know if the call is directly from the sender or if it was created by Dispatch.


Add precompile hook

Sometimes, we always want to run some code, like
rake react_on_rails:generate_packs
whenever about to run webpack
We can add a shakapacker.yml configuration so some code can run before invoking the webpack.
precompile_hook: rake react_on_rails:generate_packs

This way, "it just works" in terms of having some files ready that React on Rails needs.
Alternatives:

While we could modify the installed bin files, that can make upgrades trickier
We could wrap the bin files with new scripts.

You should already be able to do this with Rake:
# lib/tasks/assets.rake
namespace :assets do
  desc "Ensures that dependencies required to compile assets are installed"
  task install_dependencies: :environment do
    raise if File.exist?("package-lock.json") && !(system "npm install")
  end
end

Rake::Task["assets:precompile"].enhance ["assets:install_dependencies"]


A couple loop optimizations

These functions were iterating over things twice when they didn't need to, resulting in a speed decrease (however minimal). This cleans those up and makes the code easier to follow.

We're gonna need a meeting on this one.

Imma go ahead and edit this branch.


melpa-packages: 2020-03-01

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Thanks for your PR!
Since there was a merge conflict I went ahead and ran the update myself & pushed to master.

For future reference:
We have update-from-overlay which pulls from https://github.com/nix-community/emacs-overlay and creates nicely formatted commits.


Single page DSL styling

Some of these pages are big - super big. I'm not super-sold on using the bar-style headers like we have for function definitions, on the DSL pages, but they'll be okay for now.
They're also never going to be super super readable on mobile - there's just too much content. Possibly the top nav bar could include a breadcrumb tree in it, and allow a bit of navigation that way? That will require a bit more thought.
This is basically a lot of little tweaks for font sizes, margins, etc. to make things more consistent. (And getting rid of the massive "Example" headings).

Before
After

🚀 Thank you for your contribution! 🚀


AG-1395: Bump harmonized targets source version to pick up delimiter fix

Yeah ok, it looks like something about the new file did something bad to gene_info processing. What is different about the new file?

@jaclynbeck-sage I converted some semi-colon delimiters to commas, and forgot to escape them. I'll update the source and verify locally before I push up a new commit...sorry, I should have tested this locally first.

Thanks. It looks like something is... off with this file (as well as its previous versions 48 and 47 which is as far back as I looked). The raw file has 1362 lines in it, but if you read it in as a CSV either programmatically or with Excel it only has 1162 lines (versions 48-50). For v47 it as 1334 lines / 1134 lines, so a 200-line discrepancy in both versions.
Update: It looks like starting line 320, The Emory data has line breaks in the Data_used_to_support_target_selection column, which are escaped, so it does get read in successfully as one field. I guess that's ok? The raw file looks like:
320  ...,"Discovery quantitative proteomics of FrCx
321    WPCNA of multiple and consensus cohorts
322    ANOVA",...

and the read-in string looks like:
"Discovery quantitative proteomics of FrCx \n WPCNA of multiple and consensus cohorts\n ANOVA"
Is this something we're aware of/handle on the front end for display?

Thanks. It looks like something is... off with this file (as well as its previous versions 48 and 47 which is as far back as I looked). The raw file has 1362 lines in it, but if you read it in as a CSV either programmatically or with Excel it only has 1162 lines (versions 48-50). For v47 it as 1334 lines / 1134 lines, so a 200-line discrepancy in both versions.
Update: It looks like starting line 320, The Emory data has line breaks in the Data_used_to_support_target_selection column, which are escaped, so it does get read in successfully as one field. I guess that's ok? The raw file looks like:
320  ...,"Discovery quantitative proteomics of FrCx
321    WPCNA of multiple and consensus cohorts
322    ANOVA",...

and the read-in string looks like: "Discovery quantitative proteomics of FrCx \n WPCNA of multiple and consensus cohorts\n ANOVA"
Is this something we're aware of/handle on the front end for display?

@jaclynbeck-sage This seems to look ok in Agora in prod today (for the 2 genes I checked, RPH3A & STX1A): mv68 -> gene_info v56 -> harmonized targets v48

...but I could remove those /n since they are confusing and useless.
Maybe? I'm guessing the line breaks are to indicate these are 3 separate items, not one single discovery method, so the way it's displaying in Agora right now isn't right because it looks like it's all mashed together in one sentence. I think probably the \n should be replaced with commas but also there's extra spaces around the \n that need to be dealt with in order for commas to look normal.

@jaclynbeck-sage Those newlines go back to v1 of the source file from 2018 and no one has complained yet, but I went ahead and fixed it by replacing the newlines with commas, and cleaning up extra spaces.


Multiprocess application sometimes throwing UnicodeDecodeError in prometheus_client/mmap_dict.py

Hi
I have multiprocess enabled application with flask exporter, sometimes it is throwing below error
File "usr/local/lib/python3.8/site-packages/prometheus_client/mmap_dict.py", line 44, in _read_all_values
yield encoded_key.decode('utf-8'), value, pos
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 18: invalid continuation byte.
After this error my application is crashed. Can someone please help me on this?

Hmm, I haven't seen this one before, is this something that can be reproduced? Some version and environment information may also be helpful.

Hi @csmarchbanks, we are using prometheus-client 0.15.0 in docker environment

I have configured my application like below
metrics = MultiprocessInternalPrometheusMetrics(app, path="/api/metrics")
values.ValueClass = values.MultiProcessValue(os.getppid)
If I do not add "values.ValueClass = values.MultiProcessValue(os.getppid)" facing slowness in metrics endpoint


Index out of range in TravelToLastAvailableToken()

In TravelToLastAvailableToken(), when trying to determine whether a JArray has index x, an unhandled exception gets thrown if the index is out of range.

Fixed in <IP_ADDRESS>


Correct GDB Base Address

Hi,
Lately I've been trying out the debugger in Ghidra. I'm debugging a wine application, and to my surprise it works very, very well. Except for one thing; the base address gdb reports seems off. Or at least I don't understand how it got to that address.
In Ghidra, all the addresses I'm interested in start at 0x140000000. When I launch the application through the debugger and add a Watch or a Breakpoint, it is never hit. In the Modules panel I can see that apparantly gdb thinks the module is at 0x140000c00. This offsets my addresses by 0xc00 and nothing really works.
Example:
Take the address at 0x140660754 (in my Listen window). When I right click on the address and click on Watch, the address and Symbol is added to the Watch window, but at 0x140661354. (Which is exactly 0xc00 off). When I set a Watch myself, with the address 0x140660754, the value I expect is shown. So Ghidra is translating something for me, but it's wrong.
Is there any way to tell Ghidra that the binary is really at 0x140000000, and not at 0x140000c00?
Thank you very much for reading this, and of course for making this awesome tool. It's truly fantastic!
Kind regards,

This is actually a known issue, but I've been having quite the time trying to reproduce it. Wine might be a decent clue for me, so thanks for that. In the meantime, there are two known workarounds.

As you've asked, you can manually define the mappings. In the Modules window, there is a "Map Manually" button in the toolbar. Alternatively, in the Debugger menu, choose Static Mappings. Either way, you'll get the "Static Mappings" window, and you can remove and replace the incorrect mapping with one of your choosing.
You can configure the Mapper to use section information instead of module information. Without knowing your particulars, I'm not sure this will work for you, but it may. From the Ghidra Project window (not the Debugger window), select Edit → Tool Options in the menus. Then find Debugger / Workflow in the left panel. You should see an option to map sections. You'll need to re-launch your target for it to take effect.

Alright, not a problem at all! Thank you for pointing me into the direction of fixing it statically. I'm in luck because it seems the same program is almost always loaded exactly in memory as it is in the binary.
I'll try to give you some more details on my set-up, for completeness:
I'm using Wine 8.0 (Proton) as my runtime, I use winedbg --gdb $PROCESS_ID to connect, and it seems everything is fine. Ghidra is not complaining and I can pause and continue as I please. It is known that winedbg is not as fullfletched as gdb, but it does provide a gdb frontend. So I would assume the problem happens maybe because winedbg has a bit different output before it enters gdb as the frontend...
Thank you very much, at least I can continue by fixing it by hand for now!

Would anyone be able to briefly describe what causes this? I've run into the same problem, but it's hard for me to know where does this value come from just by looking at the source code.

I don't even understand why file offset is not taken into account here, I tried to follow but I got lost:
https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Debug/Debugger-agent-gdb/src/main/java/agent/gdb/model/impl/GdbModelTargetSection.java#L77

@zb3 are you unable to solve the problem using @nsadeveloper789‘a suggestions above, or is it a question of convenience or of just wanting to understand what’s going on?  As you’ve noted, we don’t support null ranges, so that could have something to do with it, or could be we’re driving the base from the section list vs. the modules list.

@d-millar, changing debugging options solves this exact problem, yet as always I'm a little curious what's going on here :)
I see that the discrepancy is introduced here:
https://github.com/NationalSecurityAgency/ghidra/blob/d932173a25bdcb36171f5c0fbaf454ba24f0435d/Ghidra/Debug/Debugger-agent-gdb/src/main/java/agent/gdb/manager/impl/GdbModuleImpl.java#L169-L173
But if 0x140001000 is the vmaStart and 0x400 is the offset, then it doesn't automatically mean the base is 0x140000c00 so again I got lost... but yeah, the workaround helps

@zb3  Just following up:  I think you're correct - if I'm not mistaken, the logic in lines 169-173, while a useful heuristic, is not guaranteed to be true.  Need to discuss this with @nsadeveloper789, but wanted you to know we're looking into this.

Yes, I had identified the same block of code in my own investigations. If I may generalize, this issue really comes down to collecting a good variety of test cases. The heuristic I applied there worked well for the Linux user-mode programs I observed on my machine :) (and it's presumably working for others'). But, it has apparently failed for some qemu system targets and now also failed for Windows user-mode targets on Wine for Linux. I assume you're also debugging with Wine? If not, I'd like to know, since it could be another useful test case.
I'm looking at two solutions, though it may turn out it's really two pieces to one solution:

Incorporate the output of info proc mappings. Potential issue here is that command is not always available, and it's not clear to me there will always be a mapping to the module's offset 0x0. I suspect I'll wind up applying a similar heuristic, but it may be more accurate.
When figuring out the mapping of the Dynamic listing to Ghidra's Static listing, incorporate the file offset information in the imported image. Since Ghidra's importer should agrees with the target in terms of memory layout, this should work. Potential issue here is that not all importers record file offset information.

If you're able to devise a patch that fixes GdbModuleImpl for your case, even if it breaks every other case, that may be useful. I realize this is a bit of an ask, but if you like to tinker....


Text double click behavior different from browser

Describe the bug
When i double click a text in an input on a web browser, the selection breaks off at period. But when i do it in the app it selects the whole text till slash. Is there a way to imitate the text selection behavior of browser on the app?
Here's how it is on the browser (same app)
https://github.com/tauri-apps/wry/assets/54323745/4e715ee2-9cca-46f7-b217-07bfe4982110
Here's what happens in the app
https://github.com/tauri-apps/wry/assets/54323745/32805a48-0d06-4b85-99ab-ebff32c41990
Steps To Reproduce

Just have an input box, write a text having period.
Open the same app in browser and notice the double click behavior difference

Expected behavior
I was expecting it to behave the same
Screenshots
Added screen recording above
Platform and Versions (please complete the following information):
OS: MacOS Sonoma 14.5
Rustc: 1.78.0
Tauri: 2.0.0-beta.22
I believe this has to do with the webview that's why i opened the issue here instead of tauri

Which browser(s) did you test this in? Did you also check Safari? That's the closest thing to tauri's webview so if it behaves the same it's a general webkit thing and not our fault.

I see, yeah Safari is the same as app but Firefox and Chrome are different. Thanks!


State control in manipulator envs

Description
With the goal of mirroring hardware robot states in simulation, an interface for directly setting the state of a robot in simulation is created and is implemented for the MuJoCo environment.
The next step towards the full mirror sim feature would be to source hardware states from the user client. The user client would send these to a local CM server, and the local sim client would pull "mirror robot state" updates from the local CM server. This indirect design is preferable for the next step (T123146501) which will require running forward simulation locally while the regular/external hardware robot client and CM are running.
Fixes # (issue)
WIP: T122948322
Type of change
Please check the options that are relevant.

[ ] Bug fix (non-breaking change that fixes an issue)
[ ] Proposes a change (non-breaking change that isn't necessarily a bug)
[ ] Refactor
[x] New feature (non-breaking change that adds a new functionality)
[ ] Breaking change (fix or feature that would break some existing functionality downstream)
[ ] This is a unit test
[ ] Documentation only change
[ ] Datasets Release
[ ] Models Release

Type of requested review

[ ] I want a thorough review of the implementation.
[ ] I want a high level review.
[ ] I want a deep design review.

Before and After
https://user-images.githubusercontent.com/12128112/185246627-81f4f0a2-5c61-4032-9be7-b84d2acb7de3.mov
https://user-images.githubusercontent.com/12128112/185246636-884f4d45-f00c-4b53-b635-e1bc537bf11d.mov
Testing
Launch a server locally using something like DISPLAY=:100 launch_robot.py robot_client=franka_sim.
Launch the example using DISPLAY=:100 python scripts/mirror_sim.py robot_client=franka_mujoco_sim
Checklist:

[ ] I have performed manual end-to-end testing of the feature in my environment.
[ ] I have added Docstrings and comments to the code.
[ ] I have made changes to existing documentation where needed.
[ ] I have added tests that show that the PR is functional.
[ ] New and existing unit tests pass locally with my changes.
[ ] I have added relevant collaborators to review the PR before merge.
[ ] [Polymetis only] I ran on hardware (1) all scripts in tests/scripts, (2) asv benchmarks.

Looks like it's on the right track.

I would also implement it for BulletManipulator (don't worry about BulletLocomotor -- this file is unused/untested)
Have unit tests that show the state is actually the set state. For PyBullet, at least, it may require some stepSimulations in order to have the state actually set.

Issue to fix numerical issues with mj set_robot_state: https://github.com/facebookresearch/fairo/issues/1354


Add openssl support

Build with option enable-openssl. By default using system installed mbedtls libraries.

I pushed a patch based on what we discussed. Thanks for the help!


[SPARK-16472][SQL] Force user specified schema to the nullable one

What changes were proposed in this pull request?
This PR proposes to force the user-specified schema to the nullable one, as Spark SQL can't validate it.
How was this patch tested?
Unit tests adde in FileStreamSourceSuite.scala and DataFrameReaderWriterSuite.scala.

Hi @gatorsmile and @marmbrus, I saw the discussion and found you are related with this one. Could you please review this?

Test build #62049 has started for PR 14124 at commit a917678.

Test build #62049 has finished for PR 14124 at commit a917678.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62049/
Test FAILed.

Test build #62053 has started for PR 14124 at commit adae8de.

Test build #62053 has finished for PR 14124 at commit adae8de.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62053/
Test FAILed.

Merged build finished. Test FAILed.

Test build #62054 has started for PR 14124 at commit 3980681.

Test build #62054 has finished for PR 14124 at commit 3980681.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62054/
Test PASSed.

val rdd = spark.sparkContext.makeRDD(Seq("{\"a\" : 1}", "{\"a\" : null}"))
val schema = StructType(StructField("a", IntegerType, nullable = false) :: Nil)
val df = spark.read.schema(schema).json(rdd)
df.printSchema()

When user-specified schemas are not nullable and the data contains null, the null value in the result becomes 0. This looks like a bug, right?

Ah, yes it seems a bug to me.. I thought it throws an exception in that case. Does this PR introduce the problem? (Just curious and to be sure).

Oh, I see, before this patch
+---+
|  a|
+---+
|  1|
|  0|
+---+

after this patch
+----+
|   a|
+----+
|   1|
|null|
+----+

FYI, currently (before this patch) the code below
val rdd = spark.sparkContext.makeRDD(Seq("{\"a\" : 1}", "{\"a\" : null}"))
val schema = StructType(StructField("a", StringType, nullable = false) :: Nil)
val df = spark.read.schema(schema).json(rdd)
df.show()

is being failed with the exception below:
Error while decoding: java.lang.NullPointerException
createexternalrow(input[0, string, false].toString, StructField(a,StringType,false))
+- input[0, string, false].toString
   +- input[0, string, false]

java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException
createexternalrow(input[0, string, false].toString, StructField(a,StringType,false))
+- input[0, string, false].toString
   +- input[0, string, false]

	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:292)
	at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
	at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$15.apply(Dataset.scala:2218)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
...

It seems unexpected behaviour anyway. I will submit a patch if this one is decided not worth being added. Thanks @gatorsmile again!

@HyukjinKwon No matter whether this PR is merged or not, I still think we should fix the above issue. Silent conversion does not look good to me.

I am a bit confused if we are allowed to read JSON (via json(jsonRDD: RDD[String]) API) with schema having fields set false in nullable.
If it is meant to be not allowed, this issue will prevents the case above.
But, yea, I think I agree that it is a potential problem anyway (even if the case above is not allowed.)

@HyukjinKwon Your patch solves this inconsistency by forcing schema as nullable at all. However,  looks like the parquet case is for compatibility, is this the same for json?

@viirya Thanks for your comment! Actually, that's I want to have some feedback for from @marmbrus .
It seems forcing to a nullable schema all is already happening when you read/write data via read/write API (but not for structured streaming and another API for json).
So, actually, the reason of this PR is, to make all consistent. The reason to make them consistent in a way that the schema is forced as nullable is what he said in the mailing list.

Sure, but a traditional RDBMS has the opportunity to do validation before
loading data in.  Thats not really an option when you are reading random
files from S3.  This is why Hive and many other systems in this space treat
all columns as nullable.

Actually, Parquet also reads and writes the schema with nullability correctly if we get rid of asNullable (I tested this before) but it seems that's prevented due to (I assume) the reason above.
@marmbrus Do you mind if I ask to clarify here please?
I think we may have to deal with this as datasource-specific problem.

Could you take a look please @marmbrus ?

gentle ping @marmbrus

@cloud-fan

What will happen if the given schema is wrong? It seems weird that we allow users to provide schema while reading the data, but without validating it.

Thanks for feedback @cloud-fan !
If the user-given schema is wrong, it is handled differently for each datasource specific.

For JSON and CSV
it is kind of permissive generally (for example, compatibility among numeric types).

For ORC and Parquet
Generally it is strict to types. So they don't allow the compatibility (except for very few cases, e.g. for parquet, https://github.com/apache/spark/pull/14272 and https://github.com/apache/spark/pull/14278)

I think so. Should we disallow specifying schemas for these?

For JDBC
it does not take user-given schema since it does not implement SchemaRelationProvider.

BTW, actually, this is not only about user-given schema.
Currently, it always reads data into dataframe by datasources based on FileFormat ignoring nullability in schema (for both user-given schema and inferred/read schema).
However, this does not happen when reading for streaming by the datasources (and another JSON api).
So, this PR tries to make them consistent to ignore the nullability in schema.

@cloud-fan If nullability should be not ignored, then I can fix this PR to make them consistent to not ignoring it (and of course I will try to identify the related problems). In this case, I will work on what @gatorsmile pointed out in https://github.com/apache/spark/pull/14124#issuecomment-231594416 about JSON (and will check the other data sources as well).
I will follow your decision.
To cut the all comments above short, (for other reviewers),

The purpose of this PR is whether it should force all schema to nullable schema or not.
This is already happening with normal reading and writing for data sources based on FileFormat.
This is for both inferred/read schema and user-given schema.
For json(rdd: RDD[String]) API, this is not hapenning.

Test build #64066 has started for PR 14124 at commit 079aae2.

Test build #64066 has finished for PR 14124 at commit 079aae2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64066/
Test PASSed.

Test build #64631 has started for PR 14124 at commit ffacb55.

Test build #64631 has finished for PR 14124 at commit ffacb55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64631/
Test PASSed.

Test build #65360 has started for PR 14124 at commit f6be52b.

Test build #65360 has finished for PR 14124 at commit f6be52b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65360/
Test PASSed.

Test build #65747 has started for PR 14124 at commit 0bc06c6.

Test build #65747 has finished for PR 14124 at commit 0bc06c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65747/
Test PASSed.

Merged build finished. Test PASSed.

Test build #67029 has started for PR 14124 at commit 3f153a3.

Test build #67029 has finished for PR 14124 at commit 3f153a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67029/
Test PASSed.

Sorry for the delay. After thinking it again, I think it doesn't make sense to allow users to specify the nullability when reading a data source. How about we turn schema to nullable in DataFrameReader.schema?

Thanks @cloud-fan, sure, that sounds great.

Oh wait, @cloud-fan, it seems, at least, Parquet files could possibly be written with not nullable fields. So, reading it without user-specified schema might also cause the inconsistency between the schema read from structured streaming and the one read from filed sources.
If you are not sure of this, I am fine with turning the schema into nullable in DataFrameReader.schema for now. Let me rebase this one first.

Test build #68474 has started for PR 14124 at commit 7306937.

Test build #68477 has started for PR 14124 at commit d240c0d.

Test build #68474 has finished for PR 14124 at commit 7306937.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68474/
Test PASSed.

Test build #68477 has finished for PR 14124 at commit d240c0d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68477/
Test PASSed.

Test build #70922 has started for PR 14124 at commit 1abaf1b.

Test build #70922 has finished for PR 14124 at commit 1abaf1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70922/
Test PASSed.

cc @liancheng , what do you think about the nullability change?

#17293 added data validation using schema information for Parquet Reader.

Let me rather close this for a while. I will reopen if it looks worth. I think the same subject could be discussed in @kiszk 's PR.


Cancelled token trading transaction stuck in processing

Soonaverse user:  0x9cf7266a665d4dc785bc37f1f5bcc9f2805ddd2f
User has cancelled a token purchase and the funds have not been returned and stuck in processing.  States that the transaction ID is:  0x9dc4495645e7a1e9708f902244b656c77a49d6a9b0b0b003bade5e2f77dcd99bf9

@flyingrabbit-lab are you able to help here?

Yes, fixed