[SPARK-8480] [CORE] [PYSPARK] [SPARKR] Add setName for Dataframe

What changes were proposed in this pull request?
Add name / setName to Dataset/DataFrame in all languages, to match RDD (continuation of #15343)
How was this patch tested?
New unit tests added for these methods.
And by hand by running in spark-shell:
scala> Seq(1, 2, 3, 4).toDS().setName("Foo Bar").cache().show()
+-----+
|value|
+-----+
|    1|
|    2|
|    3|
|    4|
+-----+

Gives this in the UI storage tab:

Thanks to @srowen for most of the work, I just passed the name through to the cache manager so that it appears on the UI. I'll look into incorporating the review comments from #15343.

Can one of the admins verify this patch?

When users are writing DataFrame/Spark SQL, will they care about RDD names?

@cloud-fan yes, this name appears in the storage tab of the UI.

@felixcheung I think I've addressed your comments from the original PR (although the similarity to names remains), but I'd appreciate you giving it another look over as I'm not very familiar with R.

thanks, @emlyn could you add R tests for this?
but before we go there, I do still think it is confusing if it is one character different from a very well established convention to set/get names in R names, especially considering auto completion rule in shell/IDE.
@shivaram what do you think? should we aim for consistency across all languages, or break away only R to avoid potential confusion?
Or, is there a more fitting names for all languages?

Yeah name vs. names could be potentially confusing -- Do we have any alternate suggestions ? As this isn't a very commonly used construct I think it might be fine to have a different name in R ?

@felixcheung @shivaram I agree it probably makes sense to rename it, what about rddName? Any other ideas?

@holdenk @davies I've adjusted the Python return pattern as mentioned in the original PR (also in rdd.py), is this OK?

@emlyn I'm not sure this should be associated with RDD, since we are working with DataFrame here?
As for the existing name methods in RDD.R - they are not public APIs

@felixcheung oh, in that case should the existing ones in RDD.R be made public (and possibly renamed)? They are public in the Python/Java/Scala APIs.

No, RDD APIs are not supported in SparkR.

If this is just for the storage tab, something like storageName could be enough ? As @felixcheung  said R users might not know what an RDD so I'd avoid introducing that in the name

As far as I'm concerned it's only to help identify entries on the UI storage tab (although others may have other ideas), so storageName sounds fine to me.

I've modified the name in R to storageName, does anyone have any opposition to that (or better ideas)?
Does it need any more tests? I'm not sure if it's possible to test that the name appears on the storage tab?

I think I've renamed to storageName everywhere now. I had to add it to generic.R too, otherwise the tests complained. I tried to guess what to put there from surrounding entries, I hope I did it correctly.
I've also added a test, which now fails with:
Failed -------------------------------------------------------------------------
1. Error: storageName on DataFrame (@test_sparkSQL.R#2608) ---------------------
unused argument (value = "foobar")
1: `storageName<-`(`*tmp*`, value = "foobar") at /tmp/spark/R/lib/SparkR/tests/testthat/test_sparkSQL.R:2608

DONE ===========================================================================
Error: Test failures
Execution halted

It also fails when running manually in sparkR:
> df <- createDataFrame(list(list(1)))
> storageName(df)
NULL
> storageName(df) <- "foobar"
Error in `storageName<-`(`*tmp*`, value = "foobar") :
  unused argument (value = "foobar")

I'm not familiar with R, so not sure how to fix it. Any hints?

Thanks @felixcheung, that test now seems to pass, although I still get Had test warnings or failures; see logs., and further up is:
Error in loadVignetteBuilder(pkgdir, TRUE) :
  vignette builder 'knitr' not found
Execution halted

Doesn't look like it's related to these changes, maybe I need to set up something in my environment?

Jenkins, test this please

Please hold on. cc @rxin @marmbrus @yhuai @liancheng @cloud-fan

Any more feedback on whether this is a reasonable way to go about this (or if not, what would be a better way)? I think some way of identifying RDDs on the storage tab would greatly improve its usefulness, it's currently very difficult to work out where any of the RDDs in there actually come from.
Not being an expert on the internals of Spark, to me this looks very similar to the name/setName methods that already exist on RDD. What am I missing that makes this case different?
Thanks.

@gatorsmile I'm probably not experienced enough to get your point but the thrust of this is simply to give DataFrames a more reasonable name in the UI. Is the toString value currently used for something where it will break if it changes?

If this is for display, I like the change. However, I am afraid it might introduce some side effects.
In this PR, the major changes I have a concern are the following two functions:
  def persist(): this.type = {
    sparkSession.sharedState.cacheManager.cacheQuery(this, Option(name))
    this
  }

  def persist(newLevel: StorageLevel): this.type = {
    sparkSession.sharedState.cacheManager.cacheQuery(this, Option(name), newLevel)
    this
  }

Before jumping to the new changes, let us see how we are using cacheQuery.
  def cacheQuery(
      query: Dataset[_],
      tableName: Option[String] = None,
      storageLevel: StorageLevel = MEMORY_AND_DISK)

cacheQuery is the API to cache the data produced by the logical representation of the given [[Dataset]]. The field tableName is used for display, but it is actually being used as an unique identifier, whose uniqueness is enfored by the Catalog. Let us show the APIs that are using the field tableName.
  override def cacheTable(tableName: String): Unit = {
    sparkSession.sharedState.cacheManager.cacheQuery(sparkSession.table(tableName), Some(tableName))
  }

  override def uncacheTable(tableName: String): Unit = {
    sparkSession.sharedState.cacheManager.uncacheQuery(query = sparkSession.table(tableName))
  }

When users calling cacheTable, we cache the result set of the logical plan of the table tableName. tableName is displayed in Storage tab as the RDD Name, but users can use the same name to get the logical plan to uncache the data.
This PR is trying to use the same field tableName for display, but users are unable to use the name to uncache the data. This looks confusing to me. I am not sure whether the others have the same concern.
IMO, a DataFrame/Dataset is like a temporary view. When we set a name to DataFrame or Dataset, we basically create a named view. If we cache it, we are putting it to a cross-session memory cache.
If we really need to improve the usability, I think we might be able to register the dataFrame/dataset as a global temporary view, or improve the existing cache management with something similar to global view management.

I like this idea, named DataFrame is actually a temp view, we can ask users to do
df.createTempView("abc")
df.catalog.cacheTable("abc")

if they want a name in the UI storage page.

Nice, all I really want is a way to identify the RDDs in the storage tab, I didn't realise it was possible with a temp view (but I think the second line should be df.sparkSession.catalog.cacheTable("abc")).

ok, are we closing this one?

I'm happy for this to be closed.

please do then, @emlyn - only the author of the PR can close the PR.
thanks for working on this.

cacheTable API doesn't allow user to change the storage level. So having a setName is useful so that we can cache the dataframe in  different storage level and still use the UI to identify the dataframe.

@phatak-dev please open a JIRA ticket for this feature request, there is no hard limit that cacheTable can't take a storage level.

The JIRA https://issues.apache.org/jira/browse/SPARK-20495 was just opened. Please feel free to submit the PR

@gatorsmile sure.

I am a Spark newbie so I might not be aware of some obvious features, but nonetheless: I consider the usefulness of a metadata field like "name" (not necessarily unique on a global/session level) for a DataFrame/Dataset object to go beyond the purely visual UI/'storage tab' use case (which apparently can be achieved through other means, like temp view).
It could be used as an identifier/selector in a collection of dataframes, either for internal 'business' logic or for integration purposes with 'outside' components that could expose those dataframes with a human-readable name if the developer/owner considered it useful to assign one (the latter does seem similar to the "temp view" use case, but here the name/label is strictly a metadata element of the dataframe itself, so there is no need to track is separately).
Actually, I see that the Dataset's as(alias) can be used just for that purpose: although it appears to be a "set-only" property (for use as 'alias' in SQL queries) and not intended to be read back, there is a fairly straightforward way of retrieving it:
http://stackoverflow.com/questions/41249806/how-can-i-retrieve-the-alias-for-a-dataframe-in-spark
However, since the "accessor" appears to be version-sensitive, it would be preferable to have a stable public  accessor for this name/alias property.

@gatorsmile gave pr for the issue.

Is there any way to create a name without caching the result? I'm parsing the Spark event logs to get some additional data about jobs and having names there is very helpful but I don't want to affect the behaviour of the application.

I am confused about this thing. If I want to see my dataframe in the storage tab, i need to do:
df.createGlobalTempView("myName")
df.sparkSession.catalog.cacheTable("myName")

Does that only give a name ?
Or is that caching it as well ?
What if prior to that I had
df.persist(....)
Will it be cached twice ?

i think its unattractive that a user had to be aware of things like global temp views and catalogs just to adjust a name of storage tab.


Adding period char to url regex in the makeHandler method.

This should fix #32.

See https://github.com/gojp/goreportcard/issues/32#issuecomment-133908992.
It appears periods are not allowed in usernames/organizations.
Also, this regex will match repo names "." and ".." both of which are not allowed, and would cause the url path to be misinterpreted anyway.

@shurcooL replied back https://github.com/gojp/goreportcard/issues/32#issuecomment-133927962
Good catch on the usernames/org names.  Also good catch on repos that simply consist of . or ...  It would probably be ideal to filter those edge cases out.
I'll look at updating the pull request to better reflect what github allows in repo names.

@shurcooL I updated the pull request to account for the cases you pointed out.  If you think there is a more clean way to filter out the . and .. case please let me know (such as part of the regex).  If not the if statement seems to do the job and is decently clean though I admit not totally ideal.

I'm really not a fan of regexes in general, so I'm all in favor of having logic moved outside as you've done. I think it's good.
LGTM.

@shurcooL the "removing binary" commit should not be separate, it was a mistake I made.  I built this codebase locally and tested it, and then I accidentally uploaded the Mac OSX binary I compiled/built for testing.
If you look at the master branch of this project, it does not have the goreportcard binary included.
Personally I think a separate PR should be created to include the default binary name goreportcard in the .gitignore file of this project so others don't repeat my mistake.

then I accidentally uploaded the Mac OSX binary I compiled/built for testing.

Ah, I see. You just removed a file you inadvertently added as part of this PR. I missed that.

Personally I think a separate PR should be created to include the default binary build name goreportcard in the .gitignore file of this project so others don't repeat my mistake.

IMO that's not necessary. It's a pretty uncommon occurrence because you typically pick which files to commit, and it's easily caught during review. I personally prefer having as empty a .gitignore as possible, so any non-native files show up when I type git status rather than being invisible and increasing risk of introducing side-effects without me being aware. But that's just my preference.
Also, in order to save time, I typically do go build -o /tmp/o (or GOBIN=/tmp/o go install) rather than go build. That way I don't need to clean up my temporary binaries. But it's just a shortcut to go build; ...; rm ./bin.

Yes, I do tend to do go build and then rm binary a lot, thanks for the go build -o /tmp/o tip.
On a side note i'm interested to hear what operating specific files or non-native files you would not include within a .gitignore that could otherwise introduce unintended side effects to a project. Also, do you then not recogmend the default .gitignore file contents github provides/suggests when starting a new project/repo?
Not trying to push back, just interested in your view point. :smiley:

I'm happy to share my view point.

On a side note i'm interested to hear what operating specific files or non-native files you would not include within a .gitignore that could otherwise introduce unintended side effects to a project.

One example I've seen that caused occasional trouble is the .gitignore that comes in in Godeps/_workspace directories, generated by godep. It ignores Godeps/_workspace/pkg and Godeps/_workspace/bin folders. Prior to 1.5, there was an issue where stale libraries could've been used when building. So there instances of people having really strange behavior, and the fix was doing rm -rf Godeps/_workspace/pkg.  I wouldn't neccessarily suggest removing that .gitignore, it's just an example.
Another example would be ignoring the binary itself (for convenience of being able to do go build, as you wanted), then doing go install and running the binary name, expecting the latest installed binary. If your PATH includes ., then the potentially old binary in cwd would be executed instead.

Also, do you then not recommend the default .gitignore file contents github provides/suggests when starting a new project/repo?

It's up to you, for all my new Go repos, I always start with a blank repository (without a .gitignore) file. It works great in my experience.
Take a look at my repositories. The only ones that have .gitignore files tend to be older C++/C#/Java projects. Those ones tend to benefit from .gitignore. With pure Go packages, I find .gitignore is not needed at all. My ideal Go package repository would have just .go files, everything else is a distraction. :P


rancher/vm-rancheros for VM does not work

Version - v1.0.0
Steps to Reproduce:

Enable VM
Try to add VM with rancher/vm-rancheros as image

Results: VM stops and won't run.  Logs say:
3/29/2016 3:51:20 PMdnsmasq: bad dhcp-range at line 2 of /etc/dnsmasq.conf
3/29/2016 3:51:20 PM/var/lib/rancher/startvm: line 99: let: NEWCIDR=16
3/29/2016 3:51:20 PM16-1: syntax error in expression (error token is "16-1")
3/29/2016 3:51:20 PM/var/lib/rancher/startvm: line 123: let: i=2886795272^(1<<16
3/29/2016 3:51:20 PM16): missing `)' (error token is "16)")
3/29/2016 3:51:20 PMError: either "local" is duplicate, or "<IP_ADDRESS>/16" is a garbage.
3/29/2016 3:51:20 PMCould not initialize SDL(No available video device) - exiting
Expected: VM should run

Is this still not working with latest fixes?

rancher/vm-rancheos hasn't been updated in forever so I doubt it works, but even if it does we still should have a modern version.

Removed the item from the dropdown in the UI for now..

Why it could't updated to RancherOS 0.4.5? Would be a great for testing and virtual host instances based on RancherVM RancherOS.
https://github.com/rancher/vm/issues/73

master 1/12
VM feature removed from Rancher. When added back in, will do a full retest.  Including retesting this bug.


rclone: 1.61.1 -> 1.62.2

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 221508 run on x86_64-linux 1

6 packages built:

celeste
git-annex-remote-rclone
librclone
rclone
restic
restique


chef server URL being set to "<IP_ADDRESS>" and fails SSL verify

I'm using the following script:
require` 'chef/provisioning/docker_driver'

chef_env = 'chefserver'
domain = 'myapp.local'
subdomain = "#{chef_env}.#{domain}"

num_appservers = 2

# Launch Application servers in parallel
machine_batch do
  1.upto(num_appservers) do |i|
      chef_server(
        :chef_server_url => 'https://chefserver.vagrant.local/organizations/chefserver',
        :options => {
          :client_name=>'admin',
          :signing_key_filename => "/etc/chef-server/admin.pem"
        }
      )
     chef_environment chef_env
      machine_options :docker_options => {
        :base_image => {
        :name => 'ubuntu',
        :repository => 'ubuntu',
        :tag => '14.04',
        },
        :ports => [443]
      },
    end
  end
end

I run this command with debug turned on:
chef-client script.rb
In debug logs I get this firstly:
[2016-03-01T16:52:07+00:00] DEBUG: Using Chef server URL: https://<IP_ADDRESS>/organizations/chefserver

Then I get these:
[2016-03-02T12:43:06+00:00] ERROR: SSL Error connecting to https://<IP_ADDRESS>/organizations/chefserver/nodes/app2.myapp.local - SSL_connect returned=1 errno=0 state=error: certificate verify failed
[2016-03-01T16:52:20+00:00] ERROR: machine_batch[default] (@recipe_files::/root/deploy_env.rb line 17) had an error: RuntimeError: Error: command 'chef-client -l debug' exited with code 1.

My client.rb file is in /etc/chef with this:
log_level        :info
log_location     STDOUT
node_name        'admin'
client_key       '/etc/chef-server/admin.pem'
chef_server_url  'https://chefserver.vagrant.local/organizations/chefserver'
validation_client_name 'chefserver-validator'
validation_key   '/etc/chef-server/chefserver-validator.pem'
ssl_verify_mode  :verify_none

I figure it's because "<IP_ADDRESS>" is the IP of the Docker container, whilst the certificate I'm providing is of "chefserver.vagrant.local".  But I can't change the chefserver URL being used.

I tried with this:
      :convergence_options => {
        :ssl_verify_mode => 'verify_none',
        :chef_server => 'https://chefserver.vagrant.local/organizations/chefserver'
      }

But it didn't change the chefserver URL. It did get me pass SSL verify because of the SSL verify mode, however I would much prefer to have this work.
Even with this, I get the following error:
                       Unexpected API Request Failure:
                       -------------------------------
                       Cannot load environment chefserver

Not sure if it'll help, but I use the following to allow ssl verification.
chef_server_cert = {
  "/etc/chef/trusted_certs/chef_my_com.crt" => {
    content: open(
      ::File.dirname(Chef::Config[:config_file]
                    )+'/trusted_certs/chef_my_com.crt'
    ).read()
  }
}
machine nodename do
  converge true
  action context.action
  file chef_server_cert
end

Been trying to copy file over to docker machine but keep getting this:
No such file or directory @ rb_sysopen - proc/7955/root/etc/chef/trusted_certs/chef_my_com.crt
It's not supposed to exist, that's my target. Something I'm not understanding about the 'file' option?

I'd like yazgoo's PR

I've written a fix for docker_transport.rb upload_file which works (FileUtils won't work here), so now I can copy certs over with "file" in my recipe.
According to 'hh' this should've sufficed, but I get
ERROR: SSL Validation failure connecting to host: <IP_ADDRESS> - hostname "<IP_ADDRESS>" does not match the server certificate
Hmm.....

I changed it to this and created PR #80, let me know if this is a valid PR else I'll close it/update it:
    def upload_file(local_path, path)
      dir = File.dirname(path)
      localfile = File.basename(local_path)
      execute(['mkdir','-p',dir])
      container.archive_in(local_path, dir, overwrite: true)
      execute(['mv',dir+"/"+localfile,path])
    end


fix pythonpath for whole-brain-segmentation

Fixes #466.
Description
As pointed out in issue #466, removing scripts from the path added in the PYTHONPATH makes the bundle work correctly.

HI @LemuelPuglisi thanks for the change.
Could you please also modify "version" and "changelog" in metadata.json (https://github.com/LemuelPuglisi/model-zoo/blob/dev/models/wholeBrainSeg_Large_UNEST_segmentation/configs/metadata.json#L3) ?
Each time an existing bundle is changed, these values also need to be updated (https://github.com/Project-MONAI/model-zoo/blob/275582718c52d75f16f791b0afbe2e88ac088107/CONTRIBUTING.md#necessary-tests)

@yiheng-wang-nv done :) Thank you for the info

/build


Bookmarks inside folders is not working as expected

Hi! 👋
I was using bookmarked to create a bookmarks file where all of them should be inside a folder. However, I think there is a problem with file generation, as bookmarks are not imported in Chrome and Edge (at least) inside a folder as expected.
Looking at the code and the Netscape Bookmark File Format specification, I think the problem is with the order of elements in this function: <DL><P> must come after <DT> and not before. I manually changed the order in a generated file and it worked as expected.
What do you think? Thanks! 😄

Perfect, thanks! Let me know if I can help in any way.


Add button for professional services

The button takes people here:
http://supportedsource.org/consulting-services-and-support/c3
Anyone who seems credible can contact us at supportedsource.org/contact to get on the list of service providers.

:+1: I'm in support of adding this, though two issues:

Could you please squash your commits?
The PNG being output from your site looks really shoddy on retina displays.

Might be asking a bit, am aware, but any chance you could make that look a bit better?
Thanks!

Sure, I'll squash the commits in a sec.
I don't have a retina screen, so I guess I never noticed it. I'll look into it though but not part of this PR. I'll let you know if/when I get a higher resolution version.

URL http://supportedsource.org/consulting-services-and-support/c3 is (no longer) reachable, serves Heroku error.

@j-schumann Good spot. Closing as now irrelevant.


Changelog_Release: Replace mvn versions:set with non mvn dependent solution

Is your feature request related to a problem? Please describe.
Currently, we pull in mvn/java just to be able to update the pom.xml with mvn versions:set. This is overkill.
Describe the solution you'd like
A simple version update in any way without using the mvn plugin. Even SED would make more sense.

Moved to https://github.com/janderssonse/changelog-tag/issues/4


transform to windows service mode

Gogs version (or commit ref): 0.10.1
Git version: 2.10.1.windows.1
Operating system: Windows 10
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[x] SQLite

Can you reproduce the bug at https://try.gogs.io:

[ ] Yes (provide example URL)
[x] No
[ ] Not relevant

Log gist (usually found in log/gogs.log):

Description
At first I run gogs by executing command ./gogs web, now I want to run gogs as windows service.
by executing
sc create gogs start= auto binPath= ""C:\gogs\gogs.exe" web --config "C:\gogs\custom\conf\app.ini""
and open http://<IP_ADDRESS>:3000 and I need to install again .
How to transform gogs to windows service mode?

I move the git.exe path from user path to system path.


feat: add load_builtin option for the loader API and add more unit test cases

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[ ] N
[x] Y

re #989

2. What is the scope of this PR (e.g. component or file name):

kclvm/loader/src/lib.rs
kclvm/sema/src/core/symbol.rs

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[x] Other

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

[x] N
[ ] Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

[x] Unit test
[ ] Integration test
[ ] Benchmark (add benchmark stats below)
[ ] Manual test (add detailed scripts or steps below)
[ ] Other

kclvm/loader/src/tests.rs

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

-25 of 51 (50.98%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 72.551%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

kclvm/sema/src/core/symbol.rs
0
3
0.0%

kclvm/loader/src/lib.rs
3
25
12.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
44137

Relevant Lines:
60836

💛  - Coveralls


Subassets on Numerics (?) #1842

Alternate discussion space to issue: https://github.com/CounterpartyXCP/counterparty-core/issues/1842.

@adamkrellenstein @reinamora137 I don't think this is the best move, let me explain.
First is that the implementation will break many current protocol assumptions, in HUGE contrast to https://github.com/jotapea/counterparty-core/issues/2 which is basically a one line change and client software is unaffected.
Then is the XCP incentives. Free subassets will ELEVATE named assets. But these incentives to spend XCP for a named (super) asset are lowered if the same benefit applies to free numerics.
Maybe I can change my opinion in the future (I doubt it lol), but at least lets not mix this one with the fee removal for named subassets.


spirv-fuzz: Add barrier instructions in dead blocks

Barrier instructions must be placed with care, because they lead to undefined behavior if executed under non-uniform control flow.
However, we can place them arbitrarily in dead blocks.

@Mostafa-ashraf19 This would be a good one for you to work on in due course.

Yes, sounds good. I suggest starting with memory barrier instructions.


Is it possible to search related data?

Hi! I would like to know if it's possible to search related data, for example:
I have a table Department and a table Employee, they are related. I'm searching for Departments and using a keyword that may only exist in some column from Employee's table, is it possible to get all Departments where there are Employees with that keyword?
Couldn't know for sure if my question is related to this one  or not.

After experimenting around, the solution was found. You must:

Relate the models (here's how)
Make the appropriate mapping in the model, using the buildDocument() function
Make it searchable (here's how)
Rebuild your mappings
Regenerate your indexes

And that's it.


Visualization of orrery body orbits

The simulation will only show that part of the orbit of the planets/moons covered in the time span we set. It would look nicer if we could draw the path for one orbit for each body. Issues

Visual - what does this mean for moons of planets?
computational - do we need to know the orbital periods? Do we fit an ellipse to a small number of points?

We could

Do this only for planets
Do this by drawing a faint (gray) circle only if the trajectory has completed less than half an orbit (angle measured to barycenter < 180 deg)


Use pip-tools to pin dependencies

Upgrades Django to the latest LTS
Specifies the ipython version (the latest version isn't compatible with Python 2)

Changes Unknown when pulling 486692da8ce74987a273d68607fcd641717764e5 on cmc333333:pip-tools into ** on eregs:master**.


postfix: removed recipient_delimiter support

The recipient_delimiter option got removed from the postfix service recently in 1 by @peti . My setup depends on this feature and as far as I can tell it never got removed from postfix nor stopped working (at least for me).
From the diff I can see that the camel case variant recipientDelimiter was used in the old service file which is wrong. All names in postfix use an underscore—therefor recipient_delimiter should be valid as described in 2. Postfix even widened its support this feature in 2.11.* as reported in 3.
I am asking for some clearance—maybe I have missed some news on the postfix side. If just the camel case was was the issue I am happy to submit a new patch to fix this.

After some testing I have submitted a patch to fix this at #8520. The camel case actually was the error. I just did not want to fast fire and re-add something that has been removed shortly without further consolidation.


Cosmetic tests

This PR does

reduce the number of unnecessary calls to data.Identifier()
cleanup remnants of fixed-bug workaround

@AkihiroSuda following comment on other PRs - some cleanup on data.Identifier


homestead: repair sed regex

I'm so sorry. i broke the plugin but now it' work correct

➜  ~ homestead --no-ansi | sed "1,/[Available|Common] commands/d" | awk '/^ +[a-z]+/ { print $1 }'
box
cloud
destroy
global-status
halt
help
hostmanager
init
login
package
plugin
port
powershell
provision
push
rdp
reload
resume
snapshot
ssh
ssh-config
status
suspend
up
upload
validate
version
winrm
winrm-config
➜  ~

Hmm I would've guessed that syntax was correct, turns out we have to escape the parentheses and or operator but that is not cross-platform compatible; sed is hilariously bad at that if we look at the rest of the answers.
There's one answer that says the macOS's sed has the option -E so that we don't have to escape the parentheses and or operator, and in my sed version (GNU) it also works, so we have two possibilities:
sed -E '1,/(Available|Common) commands/d'
sed '1,/\(Available\|Common\) commands/d'

Your version will match any one-letter that is equal to what is inside the square brackets, so it's just a coincidence that it works. Can you test the above commands and also tell me your system info?

Yes of course.
I never used sed with regex and did not suspect that they may differ from those used in php or js.
system: macOS 10.14.6 (18G87)
Last login: Sat Aug 24 00:33:46 on ttys000
➜  ~ homestead --no-ansi | sed -E '1,/(Available|Common) commands/d'
     box             manages boxes: installation, removal, etc.
     cloud           manages everything related to Vagrant Cloud
     destroy         stops and deletes all traces of the vagrant machine
     global-status   outputs status Vagrant environments for this user
     halt            stops the vagrant machine
     help            shows the help for a subcommand
     hostmanager     plugin: vagrant-hostmanager: manages the /etc/hosts file within a multi-machine environment
     init            initializes a new Vagrant environment by creating a Vagrantfile
     login
     package         packages a running vagrant environment into a box
     plugin          manages plugins: install, uninstall, update, etc.
     port            displays information about guest port mappings
     powershell      connects to machine via powershell remoting
     provision       provisions the vagrant machine
     push            deploys code in this environment to a configured destination
     rdp             connects to machine via RDP
     reload          restarts vagrant machine, loads new Vagrantfile configuration
     resume          resume a suspended vagrant machine
     snapshot        manages snapshots: saving, restoring, etc.
     ssh             connects to machine via SSH
     ssh-config      outputs OpenSSH valid configuration to connect to the machine
     status          outputs status of the vagrant machine
     suspend         suspends the machine
     up              starts and provisions the vagrant environment
     upload          upload to machine via communicator
     validate        validates the Vagrantfile
     version         prints current and latest Vagrant version
     winrm           executes commands on a machine via WinRM
     winrm-config    outputs WinRM configuration to connect to the machine

For help on any individual command run `vagrant COMMAND -h`

Additional subcommands are available, but are either more advanced
or not commonly used. To see all subcommands, run the command
`vagrant list-commands`.
➜  ~

Last login: Sat Aug 24 00:34:22 on ttys000
➜  ~ homestead --no-ansi | sed '1,/\(Available\|Common\) commands/d'
➜  ~

Ok then, let's go with -E. Doesn't work on Solaris as it seems but what does anyway...


callTelenet failed en other errors

Current situation:
As of 07-11-2024 00:05 my HA instance cannot poll the Telenet api anymore.

Remarks:

Our Telenet subscription starts at the 7th of the month, so the values are reset each month on the 7th at 00:05.
Reloading the integration results in the same error
Removing and reinstalling the integration results also in the same error

Error in the log files after reloading/installing the Telenet Integration:
This error originated from a custom integration.

Logger: custom_components.telenet_telemeter.utils
Source: custom_components/telenet_telemeter/utils.py:83
integration: Telenet Telemeter (documentation, issues)
First occurred: 07:27:50 (2 occurrences)
Last logged: 07:27:50

callTelenet failed, trying once more:

Logger: homeassistant.components.switch
Source: helpers/entity_platform.py:365
integration: Switch (documentation, issues)
First occurred: 07:27:50 (1 occurrences)
Last logged: 07:27:50

Error while setting up telenet_telemeter platform for switch
Traceback (most recent call last):
  File "/config/custom_components/telenet_telemeter/utils.py", line 81, in callTelenet
    assert response.status_code == expectedStatusCode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 365, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/telenet_telemeter/switch.py", line 26, in async_setup_entry
    await data.force_update()
  File "/config/custom_components/telenet_telemeter/switch.py", line 143, in force_update
    await self._hass.async_add_executor_job(lambda: self._session.login(self._username, self._password))
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/telenet_telemeter/switch.py", line 143, in <lambda>
    await self._hass.async_add_executor_job(lambda: self._session.login(self._username, self._password))
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/telenet_telemeter/utils.py", line 119, in login
    self.callTelenet("https://api.prd.telenet.be/ocapi/oauth/userdetails","login")
  File "/config/custom_components/telenet_telemeter/utils.py", line 103, in callTelenet
    assert response.status_code == expectedStatusCode
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:365
integration: Sensor (documentation, issues)
First occurred: 07:27:50 (1 occurrences)
Last logged: 07:27:50

Error while setting up telenet_telemeter platform for sensor
Traceback (most recent call last):
  File "/config/custom_components/telenet_telemeter/utils.py", line 81, in callTelenet
    assert response.status_code == expectedStatusCode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 365, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/telenet_telemeter/sensor.py", line 121, in async_setup_entry
    await dry_setup(hass, config, async_add_devices)
  File "/config/custom_components/telenet_telemeter/sensor.py", line 52, in dry_setup
    await data_internet._forced_update()
  File "/config/custom_components/telenet_telemeter/sensor.py", line 170, in _forced_update
    await self._hass.async_add_executor_job(lambda: self._session.login(self._username, self._password))
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/telenet_telemeter/sensor.py", line 170, in <lambda>
    await self._hass.async_add_executor_job(lambda: self._session.login(self._username, self._password))
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/telenet_telemeter/utils.py", line 119, in login
    self.callTelenet("https://api.prd.telenet.be/ocapi/oauth/userdetails","login")
  File "/config/custom_components/telenet_telemeter/utils.py", line 103, in callTelenet
    assert response.status_code == expectedStatusCode
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

I've got the suspicion Telenet has changed something, but I do not know what exactly

Thx for reporting. I see similar issue in my own sensor now, probably some changes in Telenet API. Will have to investigate...

authentication scheme changes, will require to redo the reverse engineering to simulate this new authentication scheme


fix(*): use standard app labels & selectors, add heritage label where necessary [WIP]

This PR changes all manifests to use standard app labels and selectors where appropriate. It also adds heritage labels where they're missing.
Fixes #40

(haven't manually tested this, but the code does LGTM)

I'd like to wait until after Alpha1 to merge because I am afraid there might be subtle assumptions being made in Alpha1, and I'd like to have a chance to hunt for them.

:+1:

@krancour judging on comments here, it looks like I'm the only one to have tested this. Can you also test before merge? I want to be paranoid here and make absolutely sure there are no issues that I may not have caught

I'd be happy to.

thanks!

@arschles there are several occurrences within the deis-dev chart that aren't corrected by this PR.  They probably should be.

@krancour good to know. fixing now

@krancour fixed and commits pushed. testing deis-dev now, as this PR originally predated that chart

@krancour testing shows that all services in the cluster, when launched from the deis-dev chart, have valid endpoints

I like this change btw. Makes the data in fluentd consistent

@arschles I just finished testing with both deis and deis-dev charts.  Looks good.  LGTM3

thanks @krancour. merging


printing pdf shows wrong orientation in both portrait and landscape mode and also adds white space around it

Attach (recommended) or Link to PDF file here:
_dce87b74565ae355fae88568aefb8f5e_1-Basic-Authentication(1).pdf
Configuration:

Web browser and its version: firefox Version 96.0.3 (64-bit)
Operating system and its version: fedora 34
PDF.js version: 2.12.313 (latest)
Is a browser extension: no

Steps to reproduce the problem:

open the sample pdf
try to print the pdf

What is the expected behavior : it should be in the orientation which his viewing on the pdf /screen also it should not add any white negetive space around the pdf on the print screen.
What went wrong? (add screenshot)

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

Duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=844090


Initial implementation

This PR is not finished since it is waiting for crypto-key-composer to be published.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@1133316). Click here to learn what that means.
The diff coverage is 100%.

@@           Coverage Diff           @@
##             master     #1   +/-   ##
=======================================
  Coverage          ?   100%
=======================================
  Files             ?      9
  Lines             ?     91
  Branches          ?     10
=======================================
  Hits              ?     91
  Misses            ?      0
  Partials          ?      0

Impacted Files
Coverage Δ

src/keys/__tests__/mocks/index.js
100% <100%> (ø)

src/utils/errors.js
100% <100%> (ø)

src/__tests__/mocks/index.js
100% <100%> (ø)

src/keys/ed25519.js
100% <100%> (ø)

src/index.js
100% <100%> (ø)

src/keys/rsa.js
100% <100%> (ø)

src/utils/index.js
100% <100%> (ø)

src/utils/__tests__/mocks/index.js
100% <100%> (ø)

src/algorithm.js
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1133316...d38d941. Read the comment docs.


⚠ Bofrost.be has degraded performance

In 4624d76, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1372 ms

Resolved: Bofrost.be performance has improved in 7ecc0ba after 6 minutes.


[Design System] Rename to Oui

Is your feature request related to a problem? Please describe.
Parent issue: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1766
OpenSearch Design System is long and might cause name spacing issues. However, the shorthand CANNOT just be ODS. Because there are a lot of components in OpenSearch Dashboards that are OSD. But it requires the first character to be O.
Describe the solution you'd like
Discuss the shorthand for OpenSearch Design System to replace EUI.
Describe alternatives you've considered
n/a
Additional context
n/a

Is this complete!


Feature request: support additional includes via test interface

Currently, additional includes for protobuf sources can be specified in the pact-plugin.json file, however this means that all users of the plugin must modify the configuration file for it to work. Several users have suggested that it would be best to allow this configuration to be dynamically set in the test itself:

To me, this made the plugin one dimensional per service.  It would be beneficial to be able to do this in the test itself.

and also

that’s good for now. IMO this should be a project-level configuration though.

/jira ticket

/jira ticket

0.2.4 released

Awesome, thanks! I guess we can close this now?


iOS TextField Number Input

Feature request type
Enhance component
Component name
MudTextField
Is your feature request related to a problem?
MudTextField InputType="InputType.Number" doesn't get number input pad on iOS. On Android it works but not on iOS.
Describe the solution you'd like
iOS should get a number pad input when setting InputType="InputType.Number"
Have you seen this feature anywhere else?
No response
Describe alternatives you've considered
No response
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

That worked! Thanks a lot! @Nickztar

InputMode="InputMode.numeric"


Weird indentation on <

If I do something like:
let pi_less_4 = std::f64::consts::PI < 4.0;
                                       let x = 5;

Pushing the next line to bellow the 4.0

Maybe one idea would be to change the heuristic to look at whitespace between an identifier and <.  That is, if there is whitespace, guess that it is a less-than.

We could also use a heuristic that checks for camel case, though that wouldn't help in this particular situation, or in code that allows violations of that rule.

I’ve just encountered this problem when using the bit shift left operator <<, and it ruins the indentation of the whole file below that point, everything including other functions, modules, etc.

Quick workaround: zap the lines that set < and > as parentheses, at line 94 of "rust-mode.el":
;; Angle brackets.  We suppress this with syntactic propertization
;; when needed
;(modify-syntax-entry ?< "(>" table)
;(modify-syntax-entry ?> ")<" table)

It’s a pity to lose the matching for complex types, but wrecking the indentation is a show-stopper for me.

Still a big problem comparing with namespaced constants.


Allow individually enabling salesforce/arcgis connectors

Please CR @gfiorav (I will deploy config changes before this PR)


Added redis authentication with username

Hi!
I'm going straight to the point: the current version of LuckPerms has just a password field for Redis authentication which limits the setup to a default username auth system which not always is the best option for security purposes. Then I decided to open this pull request to implement an optional username field which enables the authentication through username too. I have tried the new feature before opening this PR to test it and everything seems working correctly.

Looks really great - thanks! but do you think you could cleanup the whitespace changes? :)

Glad to see you like it. Regarding whitespace changes: do you mean the tab indents on blank lines?

You've deleted some blank lines which I guess Luck would prefer to keep

Oh nvm I didn't see the latest commit lol

Glad to see you like it. Regarding whitespace changes: do you mean the tab indents on blank lines?

Yep those! and also the changes to the imports :)

Ok, I think I've added all the blank lines back. I don't know if you have posted it already anywhere, but I think it would be really helpful have your code style configuration uploaded somewhere, all these changes "I did" were basically done by the default IntelliJ IDEA code-style config. I may like to PR more things in the future and having your code-style will avoid the need of those micro adjustments, even if you just send it privately to me, that would be really appreciated.

Looks great, thank you :)

On the topic of auto-formatters, my personal suggestion would be to disable them entirely when you're working on other people's projects, and be sure to review the diffs in the files you're editing before you commit. I find that generally there's much less friction that way. :)
However, I'm happy to share my IntelliJ config if that would be useful.
https://gist.github.com/lucko/be4287a2b08821f2a608d25ea16124c1


unable to issue certificate renewal for SAN Certificate due to failed CAA check.

Issue description
Hi There, I'm experiencing issues relative to the CAA records at the end of a certificate generation process on 2008r2, iis 7.5, win-acme <IP_ADDRESS> for a manual SAN certificate binding.
I was under the impression (perhaps false) that this was not a requirement for generating certificates, and that it is a recommended feature. As far as I understood not all DNS providers support this feature...
Any help would be appreciated.
new cert with advanced options.
manual host names
[http-01] Self-host verification files
create or update the https binding
no additional installer steps
I am running this for 48 domains + their respective www. iterations, for a total of 96 on the SAN certificate)
This only happens at the end of the check, where it is requesting the actual certificate. It's already done the validation locally.
[INFO] Requesting certificate {certificate name} 2018/8/1 16:44:36 PM
[EROR] AcmeWebException: Unexpected error
+Response from server:
+ Code: Forbidden
+ Content: {
"type": "urn:acme:error:caa",
"detail": "Error creating new cert :: Rechecking CAA: While processing CAA for {domain.tld}: DNS problem: SERVFAIL looking up CAA for {domain.tld}, While processing CAA for www.{domain}.tld: DNS problem: query timed out looking up CAA for www.{domain}.tld",
"status": 403
}
[EROR] Create certificate failed
Thanks,
Jason

I've found more information on this and I'm now closing this issue on my own.

Please inform what additional information you found on this issue

I've found more information on this and I'm now closing this issue on my own.

I'm facing the same problem, can you share your solution?

The error is most likely caused by the DNS service provider using an old software version. CAA is a relatively new record type and some older software tend to "freak out" when queried for it, instead of just replying "nope, no records of that type here". It can typically be solved by pestering your service provider or - most likely faster - switching to another one.


Solutions of MDPs

Hi,
and thanks for providing this environment to the community! I am planning to use MiniGrid for some research in regard to generalization. For this I would like to compute the deviation of learned policies from optimal one and learned Q-values from optimal ones etc.
Is there any chance to do this with the information contained in the environment?
I mean different to the ForzenLake environment the MiniGrid environments do not possess the transition probabilities.
Do you have any idea?

You could compute an approximation to the optimal policy by training several agents for a long time on environment A, and then train an agent on environment B, and ask it to generalize to environment A. Then you could look at the difference between the best policy trained specifically for A vs the policy you trained on B and transferred to A.

@maximecb Thanks for the answer! That is a nice way to circumvent the theoretical solution finding.
How complicated do you think is constructing the MDP (with this than value iteration, etc. is possible) is it for the MiniGrid environments? Especially when they are initialized randomly.

Honestly not sure. It's going to be a lot faster for smaller environments. You can also fix the starting configuration by calling env.seed(your_seed_value) each time you do env.reset() if you want.

Thanks fo the insights @maximecb !
Regarding the randomness. Is NonDeterministic a hyperparameter (to infuse randomness) or solely a description of the environment? I guess I need to make a deeper dive into the functionality of the environments.
There is something I recognized when training some agents on EmptyGrid-8x8-v0: it appears from the render videos that the agent sometimes stands still as if there is also a wait or do-nothing action. Is this the case?

The environment is deterministic but the position of the agent and configuration object can vary randomly depending on the seed for each episode.
There are actions like toggle and drop which don't apply to all environments: https://github.com/maximecb/gym-minigrid/blob/master/gym_minigrid/minigrid.py#L638


chore: comment auto-complete suggestion for native types

Or you squash merge this, and then later use the "revert PR" functionality to create a revert commit for the squash commit (I think).

Ha wow, that button really exists :D


Travis PR Builds + Travis GH-Pages Deployment = Security Vulnerability

During the course of reviewing @aaronr's last PR, #48, some issues regarding the new continuous deployment method were exposed.
Backstory:
PR #44 added tooling to run the .travis/deploy-to-gh-pages.sh build script that analyzes if the build is taking place on the master branch and will merge master into the gh-pages branch. To allow Travis to make the merge, it was given a Github OAuth token (stored privately on Travis) with public_repo scope.  The logic to determine which branch the build was occurring on was a bit incorrect, causing every build (including PRs) to be deployed to gh-pages. (This was discussed in greater detail in the comments of PR #48, and was fixed with PR #49)
Issue:
What was exposed during that process was that Travis will run whatever code it's given during a PR build. This allows for anyone to make a PR with malicious intent that will be executed with our Github credentials.  For example, someone could make a PR to alter .travis/deploy-to-gh-pages.sh so that it erases all the files within the repo, rebases the repo, and then does a forced push to Github, effectively erasing all history.  Furthermore, that same auth token could be used on any other public repo that the creator (myself) owns.  Since Git is distributed, those changes could be reversed, but it's not a situation we want to get ourselves in.

Suggestions:
@aaronr and I agreed that, for a short term fix, we should disable builds on PRs. This was done that night, immediately after the vulnerability was realized.  This means that currently, we must run tests on PRs locally, which isn't ideal.
It doesn't seem like there's any safe way to have the codebase dictate deployment steps if it involves granting every build with Github credentials.  Travis doesn't seem to have the ability to manage deployment steps from the web-admin or to only grant Github access tokens to non-PR builds.  Travis works really well for notifying us of PR's test results, so I think we should keep with using it for that.  I think we're going to have to look elsewhere for the automated deployment step.
Does anyone know of free, hosted tools that will subscribe to a Git repo and can run arbitrary node/bash scripts?  I'm going to start going through the links on this list and see which fit.

I'm thinking that having test results on PRs is better than having master auto-deploy to gh-pages, so I'm going to go ahead and re-enable builds on PRs and remove the Github OAuth Token from Travis (along with the deployment tooling). Deploying master will now be a manual process (checkout fresh master, run grunt gh-pages).

Thanks for recording this history. I'm totally fine with manually pushing to gh-pages with the grunt deploy option. Too bad that the Travis stuff could totally backfire. Great catch! :baseball:
Learning things. Unlearning things. Learning more things.


Image extraction does not handle the case when colorspace is a PdfObjRef and bmp handling is probably broken

Bug report

Description of the bug
Image extraction does not handle the case when colorspace is a PdfObjRef
bmp handling might be broken in some cases

Steps to reproduce the bug.

Tried with both pypi latest and github latest.
file:
58A_3.pdf
from pdfminer.image import ImageWriter
from pdfminer.pdfparser import PDFParser
from pdfminer.pdfdocument import PDFDocument
from pdfminer.pdfpage import PDFPage
from pdfminer.pdfinterp import PDFResourceManager
from pdfminer.pdfinterp import PDFPageInterpreter
from pdfminer.converter import PDFPageAggregator
from pdfminer.layout import LTImage

filename = '58A_3.pdf'

def get_images(layout):
    imgs = []
    if isinstance(layout, LTImage):
        imgs.append(layout)

    objs = getattr(layout, '_objs', [])
    for obj in objs:
        imgs.extend(get_images(obj))
    return imgs

with open(filename, "rb") as fp:
    parser = PDFParser(fp)
    document = PDFDocument(parser)
    img_writer = ImageWriter('.')
    rsrcmgr = PDFResourceManager(caching=True)
    device = PDFPageAggregator(rsrcmgr, laparams=None)
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    page_info = {}
    pno = 0
    for page in PDFPage.create_pages(document):
        if pno > 0:
            raise Exception('only one page expected')
        interpreter.process_page(page)
        layout = device.get_result()
        page_info = {}
        page_info['layout'] = layout
        images = get_images(layout)
        if len(images) > 1:
            raise Exception('Only one image expected')
        image = images[0]
        print(f'{image=}')
        print(f'{image.colorspace=}')
        fname = img_writer.export_image(image)
        print(f'image extracted to {fname}')
        pno += 1

Output from running the command:
image=<LTImage(Im0) 0.000,0.000,2160.000,2160.000 (18000, 18000)>
image.colorspace=[<PDFObjRef:7>]
image extracted to Im<IP_ADDRESS>00x18000.img

Output of using file on the image created
Im<IP_ADDRESS>00x18000.img: data

But, it is clear that there is a proper image in the pdf as can be seen from opening the image in any pdf viewer
adding the following change gets the image out in bmp format
--- bug_report.py	2022-05-04 13:15:27.000000000 +0530
+++ bug_report_fix.py	2022-05-04 13:22:12.000000000 +0530
@@ -6,7 +6,7 @@
 from pdfminer.pdfinterp import PDFPageInterpreter
 from pdfminer.converter import PDFPageAggregator
 from pdfminer.layout import LTImage
-
+from pdfminer.pdftypes import resolve_all, PDFObjRef

 filename = '58A_3.pdf'
@@ -42,6 +42,13 @@
         if len(images) > 1:
             raise Exception('Only one image expected')
         image = images[0]
+
+        # fix to pdfminer bug
+        if len(image.colorspace) == 1 and isinstance(image.colorspace[0], PDFObjRef):
+            image.colorspace = resolve_all(image.colorspace[0])
+            if not isinstance(image.colorspace, list):
+                image.colorspace = [ image.colorspace ]
+
         print(f'{image=}')
         print(f'{image.colorspace=}')
         fname = img_writer.export_image(image)

the output of the fixed script:
image=<LTImage(Im0) 0.000,0.000,2160.000,2160.000 (18000, 18000)>
image.colorspace=[/'Indexed', /'DeviceRGB', 255, <PDFStream(10): raw=768, {'Length': 768}>]
image extracted to Im0.18000x18000.bmp

Output of using file on the image created
Im0.18000x18000.bmp: PC bitmap, Windows 3.x format, 18000 x 18000 x 24, image size 972000000, cbSize 972000054, bits offset 54

Even though this solves one part of the problem, the resulting bmp file is still not ok.
The actual pdf looks like this:

the bmp created looks like this:

This is either 2 bugs or one bug for which i made the wrong fix.
I have decided to give up and report this.

I think I can replicate this with pdf2txt.py. This is a bug that needs fixing.
$ pdf2txt.py --output-dir ~/Desktop ~/Downloads/58A_3.pdf                                                                                22:18:24
Traceback (most recent call last):
  File "/home/pieter/.local/bin/pdf2txt.py", line 313, in <module>
    sys.exit(main())
  File "/home/pieter/.local/bin/pdf2txt.py", line 307, in main
    outfp = extract_text(**vars(parsed_args))
  File "/home/pieter/.local/bin/pdf2txt.py", line 62, in extract_text
    pdfminer.high_level.extract_text_to_fp(fp, **locals())
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/high_level.py", line 121, in extract_text_to_fp
    interpreter.process_page(page)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 992, in process_page
    self.device.end_page(page)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/converter.py", line 80, in end_page
    self.receive_layout(self.cur_item)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/converter.py", line 322, in receive_layout
    render(ltpage)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/converter.py", line 311, in render
    render(child)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/converter.py", line 311, in render
    render(child)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/converter.py", line 318, in render
    self.imagewriter.export_image(item)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/image.py", line 125, in export_image
    name = self._save_bytes(image)
  File "/home/pieter/.local/lib/python3.10/site-packages/pdfminer/image.py", line 240, in _save_bytes
    img = Image.frombytes(mode, image.srcsize, image.stream.get_data(), "raw")
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2706, in frombytes
    im = new(mode, size)
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2670, in new
    return im._new(core.fill(mode, size, color))
ValueError: unrecognized image mode

I am currently also hitting this problem with multiple PDFs. I am solving it for myself by keeping a reference list of the images in a structured output(no extraction), but I would love to be able to extract them too eventually. Do you have some pointers @pietermarsman about how to go about solving this? Anything to take into consideration while poking around?

Are you facing the first part of the problem related to pdfobjref or the second part related to bmp handling? The first part can probably be worked around with the diff i posted above.. I suspect bmp handling is a seperate issue altogether.

@TamaraAtanasoska @ramSeraph  #773 focused on the issue of BMP handling I created a pull with the fix but no one has reviewed it. https://github.com/pdfminer/pdfminer.six/pull/784
you should be able to find the fix for bmp handling in there

Thank you @ramSeraph and @pettzilla1 ! I look into both of your solutions today and see if they fix my problem, or there is more to it 😄  🙏  I hope someone reviews that PR!

It seems that I've hit problems with 'DeviceCMYK' as colorspace right after this(at least I think that is the issue), so it won't be enough to fix my problem. Still awesome to save some time on digging around about these issues at least!

Thanks @pettzilla1! Will review it soon.

As a side-note: pypdf extracts it just fine (e.g.: pdfly extract-images 58A_3.pdf, pdfly uses pypdf). Meaning you might want to have a look at how we do it :-)

@MartinThoma Thanks for the heads up. That is probably because the issue was fixed in either #773 and #784. At least I don't get an error anymore.


feat: SHIMMER - update to ghost 5

Closes #44

Superseeded by https://github.com/iotaledger/liebling/pull/57


How limit parameter works

Similar to #3.
Currently, the limit option for courses and professors seems to work like this:
planetterp.all_courses(limit="1001")
I think this would be more effective, because limit is an integer:
planetterp.all_courses(limit=1001)
Thoughts?
If we make this change, we should also change offset to be an integer.

This makes sense to me! Nice catch.


How could resnet50 be 1B parameters?

As we know, ResNet50 has a total of 25,636,712 parameters. Of these, 25,583,592 are trainable and 53,120 are non-trainable. The model has 177 layers.
Check this link for further explanation.

Closed since the leaderboard is gone...


Server error if you use an existing tag with the wrong case

If there's already a tag "ABC" in the database, and you tag a question with "ABc", the request to save causes a server error.

Nope, that doesn't seem to be the problem. Closing this because I can't reproduce it in a simple case. I'll see if it comes up again.


getProcessingTimesByStep returns a modifiable map (#29786)

Cherry pick of #29786 to fix a bug

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @jrmccluskey

Remaining tests seem to be broken on the release branch (probably waiting on Dataflow containers) so I'm going to merge and we can pick that up in validation


CoreRegistry unable to resolve a class loaded by different ClassLoader for injection

Steps to reproduce:

Module B depends on module A.
Module B has "requiredPermissions": ["perm"]
Define interface with @API(permissionSet = "perm") and a system implementing it, make sure the system has @Share(InterfaceName.class)
In module B create a system that has a field of type InterfaceName with @In annotation.

Result:
The field is not set, InjectionHelper correctly gets the class, however CoreRegistry returns null for it, because it has in its map the same class but loaded by some other ClassLoader.
Expected result:
Field gets correctly injected.

Notes to self:

Problem lives in gestalt-module
When the class loader for B attempts to get the class, first tries to get it from A's classloader. This fails (no permission) so the class is reloaded in class loader B, when we need it to get the class from the root class loader.
Believe the fix is an extra step of resolving from the root class loader if fails from parent class loader, before loading from the current class loader. Or maybe attempting load all the way up the chain?

I have fixed this issue in gestalt-module now and released as version 3.0.1, just needs Terasology to be updated to use the new version.

Digging around in old issues - engine depends on gestalt-module at 5.1.2 right now, so I expect this is resolved (although unsure if anything stable actually uses the permissionSet stuff yet)
Closing as complete.


x/tools/gopls:

gopls version
Build info
golang.org/x/tools/gopls v0.8.3
<EMAIL_ADDRESS>h1:Mxm94ix8oSARQ6svioO6SxKEYWT/VCP54/448LOHzrk=
<EMAIL_ADDRESS>h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
<EMAIL_ADDRESS>h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
<EMAIL_ADDRESS>h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
<EMAIL_ADDRESS>h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
<EMAIL_ADDRESS>h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
<EMAIL_ADDRESS>h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
<EMAIL_ADDRESS>h1:rYiuK/jn6TCxjAq1hQ6MR6pgOeP3A5gIHDxgrQ8Vbys=
<EMAIL_ADDRESS>h1:9dMzk88fnONra7zrEalqkRMGa9jMGf9B5mdzhYVyI28=
<EMAIL_ADDRESS>h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
<EMAIL_ADDRESS>h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
<EMAIL_ADDRESS>h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.14.7
go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kali/.cache/go-build"
GOENV="/home/kali/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/kali/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/kali/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/go/src/sync/atomic/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build206143841=/tmp/go-build -gno-record-gcc-switches"
What did you do?
I am a bit new to the Go Programming Language.
I upgraded Go to Version 1.81.
After upgrading, everything works perfectly fine except for this error in my VSCODE  problems pane.
undeclared name: any (compile)

I did a bit of googling and all roads seem to lead to using go mod init
What did you expect to see?
Even after running
go mod init atomic
Hoping to resolve the error, problem still persist
In fact even when I try to run go mod tidy from vscode I get this when the file is actually there.
Maybe I'm doing this wrongly.
Command 'gopls.tidy' failed: Error: no go.mod file found in /usr/local/go/src/sync/atomic.

Editor and settings
VSCODE; No setting were made relating to Go.
Logs
[Error - 15:57:04] 2022/05/01 15:57:04 DocumentSymbols failed: getting file for DocumentSymbols: no packages for non-Go file file:///usr/local/go/src/sync/atomic/go.mod (go.mod)
URI=file:///usr/local/go/src/sync/atomic/go.mod
EDIT: I've never used or opened this file(s) before, I just installed Go, opened vscode and the error appears.

Can u please run "Go: Install/update Tools" command and rebuild gopls with go1.18.x?
Wasn't there a notification to ask u to rebuild all tools? If not, that is a bug.

Can u please run "Go: Install/update Tools" command and rebuild gopls with go1.18.x?
Wasn't there a notification to ask u to rebuild all tools? If not, that is a bug.

Thanks for the responsive feedback.
There was a notification and I let it run.
Let me force it.

So I started by deleting the previously built gopls binary in my $GOPATH bin file
rm $GOPATH/bin/gopls

I also deleted staticcheck where it was previously installed on my computer so I can get a new binary built for Go 1.8.1(Although, I don't think it was necessary).
rm $GOPATH/bin/staticcheck

I also went ahead to delete the source files for gopls(I also think this too wasn't necessary).
rm -r $GOPATH/src/golang.org/x/tools/gopls

By installing gopls manually solved my problem
go get -v golang.org/x/tools/gopls

Although there could have been a better way to do this, this was what worked for me.
Thank you for your time hyangah


0.7.17 package on Hackage does not include fix for time 1.11

Hi @mgsloan,
When building the new version 0.7.17, I still got the same error as in #176 which got fixed in #175. Looking at the source over at Hackage (test/Data/StoreSpec.hs), we can see that the fix is somehow missing:
#if MIN_VERSION_time(1,11,0)
import qualified Data.Time.Calendar.Quarter as Time
#endif

whereas it should be (and is in this repo):
#if MIN_VERSION_time(1,11,0)
import qualified Data.Time.Calendar.Quarter as Time
import qualified Data.Time.Calendar.WeekDate as Time
#endif

Also, there is no changelog for 0.7.17, but that's minor.
-- Vekhir

Thanks @Vekhir !  I think I hadn't noticed that my git pull was incomplete when uploading the packages.  Uploaded new ones with incremented versions

Works now. Thank you!


Can't close full screen images

Need to add an option to go back from a full screen image.
At the moment you need to kill the application and reopen it.

Right this can be solved.
I had tried and failed miserably due to knowledge of iOS development and XCode.
Thanks to SO user coder1000 there is a solution documented in the answer here http://stackoverflow.com/questions/37204402/how-to-overlay-a-uibutton-close-button-on-top-of-a-uiwebview


Fixes an issue where URLControl checks upload permissions wrong

Fixes #5423
Summary
The URLControl tries for "ADD" permission on the selected folder, but that doesn't exist (anymore?). It changed it to "WRITE" instead, which seems logical because that's actually the permission you're controlling by giving a user or role Edit permissions on a folder. This seems to fix the issues mentioned in #5423 and in Links and Documents.
However, I did find other references to the "ADD" permission in several other places:

ResourceManager.PermissionsManager
InternalServices.FileUploadController (but that checks for both ADD and WRITE)
Modules.Export.ascx.cs

These might need changing too. Happy to do so, by the way. Just let me know.

Add permission is part of the advanced permission provider and should get mapped to write in the community provider.

Add permission is part of the advanced permission provider and should get mapped to write in the community provider.

Right. So the URLControl needs to check for ADD like it does without this PR, but the PermissionProvider needs to check for WRITE if ADD was requested by calling code?

Add permission is part of the advanced permission provider and should get mapped to write in the community provider.

Right. So the URLControl needs to check for ADD like it does without this PR, but the PermissionProvider needs to check for WRITE if ADD was requested by calling code?

yes

Add permission is part of the advanced permission provider and should get mapped to write in the community provider.

Right. So the URLControl needs to check for ADD like it does without this PR, but the PermissionProvider needs to check for WRITE if ADD was requested by calling code?

yes, that is correct

Hmm, doesn't look like that's true, because the LoadFolders method in the UrlControl uses the FolderManager which goes straight to the DB and doesn't use the PermissionProvider. Also, that looks like checks need to be somewhat more complex, like: if ADD permission exists in Permissions, then check for ADD permission, otherwise check for WRITE permission.
Any advice on how to fixed this further would be appreciated.


Responsive Web Design: Added hint to Add Borders Around Your Elements

Added hint to Add Borders Around Your Elements (https://learn.freecodecamp.org/responsive-web-design/basic-css/add-borders-around-your-elements)

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

LGTM! Thank you @greggubarev !


Create setup.py

I wanted to be able to download this as a library with pip, and this made it much easier. This should help people to install this as a library using the command:
pip install git+https://github.com/Shaofanl/RaspberryPy.git

Thanks MiraWelner! Merged


WIP:add generated type annotations

Add automatically generated annotations.
Generated by pytest-annotate.
Commands used:
pytest --annotate-output=./annotations.json
pyannotate --py3 -w --type-info ./annotations.json spoonbill/

Seems like main issue is circular imports, I wonder what is the easiest way to overcome it.


Refactoring to the context of SafeFor(H)All

Codecov Report

Merging #45 (494d9b9) into master (e485b63) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master      #45      +/-   ##
============================================
+ Coverage     72.15%   72.24%   +0.09%
  Complexity      399      399
============================================
  Files            70       70
  Lines          1232     1236       +4
  Branches        125      125
============================================
+ Hits            889      893       +4
  Misses          311      311
  Partials         32       32

Impacted Files
Coverage Δ

src/main/java/safeforhall/AppParameters.java
75.00% <ø> (ø)

src/main/java/safeforhall/Main.java
0.00% <ø> (ø)

src/main/java/safeforhall/MainApp.java
0.00% <ø> (ø)

src/main/java/safeforhall/commons/core/Config.java
68.18% <ø> (ø)

...ain/java/safeforhall/commons/core/GuiSettings.java
69.23% <ø> (ø)

...main/java/safeforhall/commons/core/LogsCenter.java
78.37% <ø> (ø)

...c/main/java/safeforhall/commons/core/Messages.java
0.00% <ø> (ø)

...rc/main/java/safeforhall/commons/core/Version.java
82.50% <ø> (ø)

...ain/java/safeforhall/commons/core/index/Index.java
100.00% <ø> (ø)

...ll/commons/exceptions/DataConversionException.java
100.00% <ø> (ø)

... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e485b63...494d9b9. Read the comment docs.


File input append

Description
Current version only performs actions on files added in most recent add file action as it removes files from file input. Removed that line
Closes #()
Checklist

[ X] I have read the Contribution Guidelines
[ X] I have performed a self-review of my own code
[ X] I have attached images of the change if it is UI based
[ X] I have commented my code, particularly in hard-to-understand areas
[ X] If my code has heavily changed functionality I have updated relevant docs on Stirling-PDFs doc repo
[ X] My changes generate no new warnings
[ X] I have read the section Add New Translation Tags (for new translation tags only)

it was removed in the PR https://github.com/Stirling-Tools/Stirling-PDF/commit/4998ad064a115835fbbf7cb745b81017eb33559e and now added again, which way should be taken?
@omar-ahmed42 @reecebrowne

it was removed in the PR 4998ad0 and now added again, which way should be taken?
@omar-ahmed42 @reecebrowne

Omar correctly updated to append rather than set but the removal logic is incorrect

Hello, there was an issue that I noticed in img-to-pdf in "merge" mode that resulted in duplicate files being present in form data that's why I added that line

That is now handled elsewhere

That is now handled elsewhere

Oh, you're right I didn't know you fixed it elsewhere, Great work! 👏🏻

@omar-ahmed42 Sorry mate you were quite right. There was a similar issue that is handled elsewhere but the whole of this downloader logic doesn't work with some listeners that are used elsewhere, My fix has fixed one symptom and yours has fixed another but both aren't getting to the root of it. Closing this for now as it's unearthed a larger potential issue.


handle the case in which user set email as private in github

$ pylint fastapi_sso/
************* Module fastapi_sso.sso.base
fastapi_sso/sso/base.py:80:9: W0511: TODO: Remove use_state argument and attribute (fixme)
************* Module fastapi_sso.sso.microsoft
fastapi_sso/sso/microsoft.py:25:35: W0511: TODO: Remove use_state argument (fixme)
fastapi_sso/sso/microsoft.py:34:35: W0511: TODO: Remove use_state argument (fixme)

Your code has been rated at 9.90/10 (previous run: 9.89/10, +0.00)

Hi @tomasvotava ,
I'm sorry but I have been busy in the last 3 weeks.
Finally, I've made the PR about #54 and retested it in one of my projects.
I've some problems with the coverage and code quality, can you help me ?

Resolved in 0.9.0


Automatic retrival of factorie version in bin/fac

Closes factorie/factorie#363.
I have updated bin/fac to automatically retrive the factorie version from pom.xml

This breaks on release builds as they aren't of the form d.d-SNAPSHOT. Also if any of the dependencies ever get a version number of the form d.d-WORD then it will pick those up too which might cause problems.

@Craigacp ya. I have fixed it and now I capture any version format of the type d.d(-WORD)* which should work for most cases. Also, I only pick the version information available in the first match so as long as the factorie details are the first match it should work.

Nice, thanks


The Reactor: A Sample-Efficient Actor-Critic Architecture  [1704.04651]

The Reactor: A Sample-Efficient Actor-Critic Architecture
https://arxiv.org/abs/1704.04651

Now the ICLR 2018 version of the Reactor paper has a lot of updates and new impressive results, replicating it is more important than before.


Fix translation FR

Description

Fixed

Fix translation 🇫🇷

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I wrote unit tests for my code (if tests is required for my changes)
[x] I have made changes in CHANGELOG.md file

:+1: Great thanks for your contribution!


Help catching exception?

Hi!
We have some code that is trying to read nexml from treebase. When there are problems with the nexml we want to try to catch the error in a friendly way - but we are having an issue where the whole nexml is output to stderr.
e.g.:
import sys
from dendropy import DataSet
url = "https://raw.githubusercontent.com/TreeBASE/supertreebase/master/data/treebase/S11681.xml"
try:
           dna = DataSet.get(url=url, schema="nexml")
except Exception as ex:
            sys.stderr.write("\nERROR: Problems reading the input data from treebase:\n")
            message = str(ex)
            sys.stderr.write("DENDROPY ERROR:'"+ message + "'" +'\n')
            if "protein" or "amino" in message:
                sys.stderr.write("\nIt appears this may not be a DNA alignment - physcraper can only use DNA currently.\n")
            sys.stderr.write("Investigate, and potentialy download alignment directly at \n{}\n".format(url))
            sys.exit()

Any advice?

Modified in c0aaf6c6.

Awesome, thanks!


Remove ristretto255 trait implementations

I noticed today that the curv traits are implemented for ristretto255.
ristretto255 is an abstract group, not an elliptic curve. See the ristretto255 RFC draft for more details. The curv traits provide APIs for elliptic curves, and thus it is a type error to implement them for ristretto255.
I briefly looked at the trait implementation, and found an example of where the type error manifests as an implementation error:
https://github.com/KZen-networks/curv/blob/b326b2708560eaa0e8fd86f2c2d7d20d6eafbdce/src/elliptic/curves/curve_ristretto.rs#L278-L289
Besides the fact that ristretto255 elements do not have coordinates, this code is parsing an encoded ristretto255 element (which includes the representation of a field element modulo p = 2255 - 19) as a ristretto255 scalar (which are field elements modulo ℓ = 2252 +<PHONE_NUMBER>7372353535851937790883648493), which is a type error, and returns a scalar that has been reduced from a field element, losing information.

Hey @str4d, thanks for looking into the code!
For Ristretto we are using curve25519-dalek which I belive is an instantiation of Ristretto for curve25519.

curve25519-dalek is a library providing group operations on the Edwards and Montgomery forms of Curve25519, and on the prime-order Ristretto group.

What am I missing?
About the x/y coordinates - I agree, there's an error there. In practice we never use the coordinates so I think we will make it unimplemented as well. However, I don't see how it relates to your first point?

For Ristretto we are using curve25519-dalek which I belive is an instantiation of Ristretto for curve25519.

curve25519-dalek is implementing ristretto255 as specified in the RFC draft. It happens to use the Edwards form of Curve25519 as its internal representation, but that is an internal implementation detail, not exposed as part of the Ristretto API.

curve25519-dalek is a library providing group operations on the Edwards and Montgomery forms of Curve25519, and on the prime-order Ristretto group.

What am I missing?

Exactly as that documentation says, it provides group operations on the Curve25519 elliptic curve, and on the prime-order Ristretto group.
I strongly recommend reading the ristretto255 RFC, but quoting two relevant sections:

Ristretto implements an abstract prime-order group interface that exposes only the behavior that is useful to higher-level protocols, without leaking curve-related details and pitfalls.

ristretto255 is an abstraction which implements a prime-order group, and ristretto255 elements are represented by curve points, but they are not curve points.

In short, ristretto255 not an elliptic curve, and therefore the ECPoint trait should not be implemented for it, as that trait is intended for elliptic curves (as documented in this crate). A more generic group trait (such as the one I am currently working on for the group crate) would make sense for both ristretto255 and the various elliptic curves, but users of the ECPoint trait can reasonably assume that they will only be given elliptic curves, not generic groups.

I see your point. You are concerned because we named the trait in this library ECPoint, is that right?
This is probably a bad naming issue because this trait aims to offer an interface for generic group element (I admit that it might not be the most accurate set of operation required but this is what we needed for our generic ECC [put aside the issue with coordinates for a second]).
When you use this outside this is the names you see:
pub type GE = Secp256k1Point;
pub type FE = Secp256k1Scalar;

If we were to rename ECPoint to something more generic over groups, would it solve the issue?


BigQuery Partition Generation

This is such a big PR and I've looked at it so many times that I don't think I can add much value by looking at it again. I think it's in a good enough state to try out, I'm sure we're going to run into some issues when we use this code in production but we can handle those with smaller, more focused PRs. We've also discussed how some of the tests can be improved with some refactoring but will leave that for another PR.


Map critical memory regions before loading kernel if detected

I'm wondering if its possible to explicitly map memory regions in the bootloader? For example, could we detect to see if the APIC is available, then map its entire memory region into the page table before loading the kernel, and then let the kernel configure the IDT? We can even pass the address of the APIC region (its default is FE0000, if I'm not mistaken, but this can change) in the boot information structure. Is this feasible, and if so how would it be accomplished?

What would be the advantage over doing this in the kernel? We try to keep the bootloader as small as possible and do only the work that would be impossible or much more difficult to do after the load. Detecting and mapping the APIC sounds like something that can be done in the kernel, or am I missing something?

I've tried to do that myself in my own kernel. For some reason its page-faulting through when trying to read memory address FEEB0h (that might just be a bug I can't seem to find though). Strangely enough, it happens immediately after I enable interrupts (by that point I've configured the PIC completely). According to OSDev I'm supposed to initialize the PIC whether an aPIC is present or not. My code is available at https://github.com/ethindp/kernel/blob/master/src/interrupts.rs.
I know I do some of the initialization sequences twice (or thrice) and I could probably trim this down by quite a lot, but its a (supposedly) good working implementation to start with. But this page faults when it hits line 115. According to GDB, this page faults as soon as it tries writing "INTR: Stage 1 initialization complete" to the serial port. Though this would normally suggest a bug in the uart_16550 crate, this doesn't make sense because that works until it reaches this point and then it dies. So that's why I suggested this. (The title is a bit misleading, sorry about that.) I'm not really sure how to fix this page fault; it doesn't seem to (necessarily) be a problem with my code -- not that I can find at least.

Unfortunately I'm pretty busy right now, so that I don't have the time to help you debugging this.
I still see no real advantage of including the APIC initialization in the bootloader instead of doing it in the kernel, apart from convenience.

This issue is quite old and I solved it a while ago. Closing.


[Feature Request] NavbarSection, Box, Flex, Collapse Components

Have loved playing around with the AppShell component that you've introduced into this library as it makes structuring my applications much cleaner. Would love if you could add in a wrapper for the Navbar.Section component at some point as without it is a bit of a pain to use the Navbar component and removes a bit of the readability of my code. I also think that having access to the Box component would make the MediaQuery component more useful. And finally, the addition of the Flex and Collapse components should round out all of the components necessary in order to create pretty advanced dashboard layouts. I might try to see if I have some time next weekend to poke around in your code and see whether I might be able to help on wrapping some of these components myself. Cheers!

Is there any progress on collapse components, especially a collapsable card? :pray:

Collapse element like dash_bootstrap_components would be nice

These components have all been added in the upcoming version of dmc.