[metadata] Add RocksdbMetadataStore

Motivation
Add Metadata Store implemented with rocksdb.
This is part of the work to remove zk from standalone server. All metadata can be stored in rocksdb.
Modifications
Add Metadata Store implemented with rocksdb.
Verifying this change

[x] Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as each test class extends BaseMetadataStoreTest
This change added tests and can be verified as follows:

org.apache.pulsar.metadata.impl.RocksdbMetadataStoreTest

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] no-need-doc

Not ready for user yet.

/pulsarbot run-failure-checks

@eolivelli

RocksDB is only local, so you could use this mostly only for Pulsar Standalone.

Yes, that's plan. This is one of multi-step work for Pulsar Standalone.

Why should I use RocksDB and not ZooKeeper for Pulsar Standalone ?

IMHO, in Pulsar Standalone, we already have RocksDb in dependency, so by removing zk from standalone mode, we can make it more lightweight, with less running memory consumption and less time of starting up.  Although pulsar standalone is not for production use, it's quite useful and important for offline developing, testing and CI/CD.

Also we should state clearly that using this implementation is not possible for production usage because each broker will have its own copy of the data, as RocksDB is not a distributed database.

Yes, more detailed doc will be added later, since there is some more work to achieve this, like "Bridge BookKeeper metadata into MetadataStore"...
@merlimat @codelipenghui @hangc0276 Do we have more detailed doc for this (Removing zk from Pulsar standalone)?

@eolivelli

RocksDB is only local, so you could use this mostly only for Pulsar Standalone.

Yes, that's plan. This is one of multi-step work for Pulsar Standalone.

Why should I use RocksDB and not ZooKeeper for Pulsar Standalone ?

IMHO, in Pulsar Standalone, we already have RocksDb in dependency, so by removing zk from standalone mode, we can make it more lightweight, with less running memory consumption and less time of starting up. Although pulsar standalone is not for production use, it's quite useful and important for offline developing, testing and CI/CD.

Also we should state clearly that using this implementation is not possible for production usage because each broker will have its own copy of the data, as RocksDB is not a distributed database.

Yes, more detailed doc will be added later, since there is some more work to achieve this, like "Bridge BookKeeper metadata into MetadataStore"...
@merlimat @codelipenghui @hangc0276 Do we have more detailed doc for this (Removing zk from Pulsar standalone)?

Why we need to remove zk from pulsar standalone. It can be an option.
I can still use pulsar standalone to test some metadata storing on zk.

Why we need to remove zk from pulsar standalone. It can be an option.
I can still use pulsar standalone to test some metadata storing on zk.

Yes, it's optional.

RocksDB is only local, so you could use this mostly only for Pulsar Standalone.

Yes, that's the only point of it.

Why should I use RocksDB and not ZooKeeper for Pulsar Standalone ?
Starting RocksDB vs Starting ZooKeeper is not a big difference.

Not really, there is a huge difference in performance and, much more important, in resources usage.
Pulsar in Standalone mode is not a distributed system, so it doesn't need any of the feature that ZK provides.

Why we need to remove zk from pulsar standalone. It can be an option.
I can still use pulsar standalone to test some metadata storing on zk.

For general users, it's much better to not use ZK for standalone.
The primary role of standalone is not being a (Pulsar) developer tool.

Also we should state clearly that using this implementation is not possible for production usage because each broker will have its own copy of the data, as RocksDB is not a distributed database.

One can also use Standalone for production, in determined environments, so the distinction here is not very meaningful.
The distinction is : distributed vs standalone

@merlimat We already have a metadata interface, so I think add rocksdb as an option is not much difference with remove zookeeper. If people who use zookeeper standalone for production, they can still use zookeeper standalone. Or they have to migrate the metadata from zookeeper to rocksdb. What do you think?

@merlimat We already have a metadata interface, so I think add rocksdb as an option is not much difference with remove zookeeper. If people who use zookeeper standalone for production, they can still use zookeeper standalone. Or they have to migrate the metadata from zookeeper to rocksdb. What do you think?

That's outside the scope of this PR and a full proposal would have to be submitted for this.
My short form opinion is:

If a standalone Pulsar is already there, keep using the existing format with ZooKeeper
If starting a fresh Standalone, default to RocksDb for metadata and Functions state store. Also, use local FS for storing functions jars
If user configure to use ZK, keep using ZK.

@eolivelli Any suggestions about this work?

@eolivelli PTAL again

I just have a question. When we use RocksDB as Pulsar standalone's metadata store, how do we check the metadata like we did via bin/pulsar zookeeper-shell?

I just have a question. When we use RocksDB as Pulsar standalone's metadata store, how do we check the metadata like we did via bin/pulsar zookeeper-shell?

There is a PIP for this. See #13346

Follow up with docs update in #13830.


Can't change the font size

You can change a bunch of properties of the control, but you can't change the font size or icon size.

Hi, thank you for pointing to these features, Will add these to my Todo list, so i implement dynamic style in the next version.


Get-Secret fails to retrieve some secrets listed by Get-SecretInfo

This should speak for itself:
PS❯  get-secretinfo | ? { $_.name -eq 'LeCab' } | select name,type

Name          Type
----          ----
LeCab PSCredential

PS❯  get-secret -name LeCab
Get-Secret: The secret LeCab was not found.

In the affected vault, Get-SecretInfo lists 85 secrets, this failure occurs for 8 of them:
PS❯  get-secretinfo | ? { $null -eq (get-secret -name $_.Name -ea 0) } | select name, type

Name                                        Type
----                                        ----
Enquête "Reconnaissance au travail" PSCredential
FileLocator Pro [LITE]              PSCredential
International SOS                   PSCredential
LeCab                               PSCredential
OgoneVisaNo3DS                      PSCredential
Pick Go Pay APN key                 PSCredential
Samsung Galaxy A40                  PSCredential
Windows 7 Ultimate                  PSCredential

I'm running PowerShell 7.1.4 on Windows 10.0.19043.1165.amd64fre.vb_release.191206-1406 with Microsoft.PowerShell.SecretManagement version 1.1.0 and SecretManagement.KeePass version <IP_ADDRESS>

@sba923: It looks like a similar issue I run into.
If you do store an item in Keepass with no (or empty) password, SecretManagement.KeePass would internally return a null-value. This finally seems to be treated as "no secret found" by MS.SecretManagement.
IMHO this is OK. Because afiak the SecretManagement module would generally not allow to set "empty" secrects.

@danubie thanks for sharing your thoughts
I actually use KeePass to store all kinds of secrets, some of which have an empty "Username" field and/or an empty "Password" field, for instance for software product keys, phone IMEIs....

That's exactly where I got caught: A folder for API keys having "usernames" but no password

Hi,
I just put the API key in the password and a "1" for the username


add a new blog of Nvidia

Check the preview: https://deploy-preview-4652--daocloud-docs.netlify.app/en/blogs/20240619-nvidiaai


Containerize the Fermyon compute unit?

I am going to disguise a proposal in a question. It looks like you are choosing a VM (or a laptop OS) as the compute unit for running the Fermyon platform. I don't know if this a choice you have made for initial practical reasons or for more strategic reasons in terms of where you want to bring the tech. However, the more I play with Wasm (and Spin and now the Fermyon platform to an extent) the more I find it way more convenient to package the compute unit as a container image (and execute it as a container). First and foremost there appears to be dependencies with things like spin that are easier to solve with a container than a VM (e.g. I can't run spin on AL2 due to the glibc_2.27 requirement). There seems to be practical requirements (https://github.com/fermyon/installer/issues/45) that would be easily resolved with a container sandbox. A containerized setup would also provide "for free" a modern horizontal scaling mechanism (https://github.com/fermyon/installer/issues/63). A containerized deployment would give the user more options to deploy the Fermyon platform (docker compose locally, ECS/Fargate on AWS, Nomad for a more infrastructure agnostic deployment mechanism - Nomad supports containers in addition to Linux processes so you don't necessarily need to ditch Nomad from your stack).
Right now you seem to have a series of bash scripts you run to install locally (or as user data for setting up on EC2). What I am thinking about would be to turn all that logic into a dockerfile to build images that you could then (way more) easily instantiate locally or in the cloud with minimal IaC differences. In the past few weeks I was able to build a small PoC of a containerized version of Spin running on ECS/Fargate that worked very well and there would be no reason why one couldn't be able to run the whole Fermyon platform on the same stack if there is that appetite.
Everything I have said would also apply to K8s (but I am not sure if K8s is the platform you see the Fermyon platform users wanting to deploy on).
Is this move from a VM compute unit to a container compute unit something that you are considering? Or does it crash with how you see Wasm evolving over time where the VM (the entire OS) should be the shell where you run Wasm code? There is quite a bit of philosophy into this question/proposal, I know :)

Thanks for the issue and discussion points, @mreferre. Indeed, as you've seen (https://github.com/fermyon/installer/issues/45 and others), we're continuing to explore approaches that lower the barrier of entry for running the Fermyon platform. Wanted to chime in with a little background and then hopefully the discussion can start to grow around the thoughts/suggestions you've mentioned.

I don't know if this a choice you have made for initial practical reasons or for more strategic reasons in terms of where you want to bring the tech.

A little bit of both. Nomad, being our scheduler of choice for the platform, is intended to run directly on the host. In adding the various other moving bits on top of this foundation (i.e. Bindle, Traefik, Hippo), all in the form of Nomad jobs, we stuck with the same approach (either installing them onto the host directly or having Nomad fetch binaries as needed). There is definitely a strategic reason, at least in one main area, for this approach as well: to harness the core benefits that we hope to deliver with the platform (speed, efficiency, scale-to-zero), we intend for the Spin apps themselves to run on the host (i.e. on a registered Nomad worker) directly.
All of that being said, we're definitelly open to exploring ways in which containerization might help users wishing to run the platform have a better go at it. Although the effort to run Nomad itself in a container doesn't appear immediately trivial, the other platform infra components (Bindle, Traefik, Hippo) and perhaps even Spin itself (i.e. the spin process running a given app) could certainly be eligible, as long as the platform (and the value it provides) isn't adversely affected by the introduced complexity and/or overhead of containerization.

Thanks @vdice for the background. I love this discussion.
I should have made it more clear that my appetite for using containers in this very specific use case is very much geared towards a better packaging mechanism and not so much around "containers for microservices". In other words I see containers being used here more like "glorified VMs" (here I said it!). It just feels to me that attacking your operational challenges discussed above with OCI images and Dockerfiles is easier than doing so with AMIs and user_data (I am trivializing it). So yes in a way it's about introducing another compute abstraction (containers on top of VMs) but I sense that the outcome is net positive and the risks of "the platform (and the value it provides) isn't adversely affected by the introduced complexity and/or overhead of containerization" are minimal and/or dwarfed by the operational benefits that containerization can introduce.
What I was envisioning was containerizing Bindle, Traefik, Hippo, Spin and every other process you would need to run so that these containers can be scheduled just like how you are scheduling today the binaries on the EC2 instances.
In the past few weeks I have started to play with Spin (prior to the platform announcement) and after playing with a VM deployment I found it so much easier to package everything into a container and use it that way. This is the Dockerfile I came up with (far from being optimized):
FROM ubuntu:latest

# this ARG is required for cmake not to prompt for a geo
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt install curl jq git cmake zip -y

SHELL ["/bin/bash", "-c"]

RUN LATEST=$(curl -s https://api.github.com/repos/fermyon/spin/releases/latest) \
    && X86URL=$(echo $LATEST | jq -r '.assets[].browser_download_url' | grep linux-amd64.tar.gz) \
    && X86ARTIFACT=$(echo $LATEST  | jq -r '.assets[].name' | grep linux-amd64.tar.gz) \
    && curl -L -O $X86URL \
    && tar -zxvf $X86ARTIFACT \
    && mv spin /usr/local/bin/spin

RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup target add wasm32-wasi

EXPOSE 3000

# this assume that files/artifacts are mounted on the /wasm folder
CMD spin up --listen <IP_ADDRESS>:3000 --file /wasm/spin.toml

This is a super small PoC I put together off the back of the above container where I would launch a couple of AWS Fargate tasks connected to a shared EFS volumes mounted on /wasm/:

The CodeBuild project just builds Wasm artifacts and place them in a shared EFS volume that the 2 Spin containers go grab. By the way this is why I asked for hot reload support in Spin here and Arm support here (Fargate ships with support for Linux/x86, Linux/Graviton and Windows/x86).
When I saw the Fermyon platform announcement I have seen similar operational issues that I have observed with Spin.I intended to play around the individual pieces and containerizing them just like I did with Spin but I haven't had time to do so just yet (it also feels a bit more challenging to stitch together 4-5 pieces rather than just one and I did not want to end up in an "experiment cul-de-sac" if you were not open to go down this route).
I would really be happy to collaborate and build a PoC together on the scenario above (if you feel like it).

@mreferre very cool, thank you for sharing!

What I was envisioning was containerizing Bindle, Traefik, Hippo, Spin

I think images for these would definitely prove useful to developers more comfortable with the container workflow (and/or as an alternative mechanism to setting up their host machine directly).  As you've mentioned, the applicable Nomad jobs can be updated to use the docker driver.  If this provides a convenient (and/or more approachable) way for users to test out and try the platform, this will be excellent!  This might be a good start to explore how the platform runs/performs using this technology.
We're definitely always curious to learn about the creative ways engineers come up with to use and run Spin, such as the intriguing AWS setup you've illustrated.  If you'd be up for it, I bet it would be a welcome demo in the Spin Developer's Meeting!


I met some problems when I install TensorRT.

【ERROR: tensorrt-8.6.1-cp311-none-win_amd64.whl is not a supported wheel on this platform.】
I don't know how to sovle this error.

PS:
My computer is Win11 (23H2) x64，python=3.11.6

Try python -m pip install --upgrade pip


curate the anxiety paper

See https://github.com/Journal-Stat-Data-Science-Education/JSDSE-volume31/tree/main/jsdse_anxiety
The goal is to make the existing code reproducible and match the output from the published paper.

@nicholasjhorton track down which paper this is

let's hold on this for now


building the source code to be used programatically

Before the only transpiled code we were generating was the one used for the cli.
We are in need of using codeowners-generator programmatically so we decided to add a new dist in the package (build/) to transpile the source code as well.

Pull Request Test Coverage Report for Build 335027976

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 327625988:
0.0%

Covered Lines:
197

Relevant Lines:
197

💛  - Coveralls


Documentation updates, some more refactoring, and project updates

Good call @karimnaaji - issue here https://github.com/mapzen/on-the-road_ios/issues/25


Improve parsing of float values returned by API

This change improves the parsing of float values returned by the API. This is done by using the numerical value, scale factor and number of decimal places specified in the API response instead of trying to cast the string representation directly to a float. This is only done for values that can be identified as floats by their unit.
Casting the string representation directly to float caused problems when the string contained , instaed . for decimals points.
I have also mapped the units returned by the API to the corresponding SensorDeviceClass units in Homeassistant.
This PR should resolve issues #1 and #2

@civ0 I´ll install the version of your PR on my friends heating this week.
In case somebody could give me a quick overview on how that integration (respectively HA integrations in general) work(0)s I could also do a code review.

@Vinzent82 Thanks for trying it out. I already noticed that updating an existing installation of the integration does not give access to more entities, because th available entities are only generated during the first installation. I am currently trying to fix this.
I also don't have any prior experience with HA integrations, so I am not able to give a proper introduction to this topic.

@civ0 ok at least I can tell you that I was able to install your branch successfully and it works well.

@civ0 Thank you for your work.
Unfortunately I haven't had time to look into your pull request earlier...


db_sync: correct logic, do not ignore failures

Simplify and correct logic to not ignore errors which may occur when
checking the db_version using the manage tools.
Also remove no longer needed exit_status module.

Looks good, although the haproxy fix, didn't I merge that in a different PR?

Looks good, although the haproxy fix, didn't I merge that in a different PR?

Yea, I'm not sure how that made it into my commit here too. Removing that extra file.

Yea, I'm not sure how that made it into my commit here too. Removing that extra file.

Rebased, squashed and remove the extra haproxy handler.
Thanks for catching that and calling it out!

Rebased, squashed and remove the extra haproxy handler.
Thanks for catching that and calling it out!

I'm so sorry!  I just noticed this. I think it would be better if we sync the db prior to starting the service. I've seen some services that will actually create/migrate the database upon first connection if it isn't right, or the service will crash and be in an odd state if the database doesn't have the tables its looking for. I just think it'd be safer if we sync first, then start the service. Thoughts?
Otherwise this looks fantastic1

I'm so sorry!  I just noticed this. I think it would be better if we sync the db prior to starting the service. I've seen some services that will actually create/migrate the database upon first connection if it isn't right, or the service will crash and be in an odd state if the database doesn't have the tables its looking for. I just think it'd be safer if we sync first, then start the service. Thoughts?
Otherwise this looks fantastic1

Ah, that was a good call and looking at the code definitely explains the intermittent failures I would see with the glance image upload and sync.
Should be clean now!

Ah, that was a good call and looking at the code definitely explains the intermittent failures I would see with the glance image upload and sync.
Should be clean now!

:rocket:


Feature/get pull request patch

The patch target, much like the diff target, is accessible just by changing the header.

This PR, among others, keeps me stuck in a fork.  Assuming I update so that it applies cleanly is there anything else needed to get merged?

@TomMD thanks for the ping. Yes, I'll go through the PRs and make new release, let's see if this applies cleanly as is.

Superseded by #350


Sending Messages and Not Seeing Them in the Chat

I logged into my account and sent some messages to a chat. Although the response body returns a status code of 200, sometimes I can't see the message in the chat. However, after waiting a bit and sending the message again, it does appear.
I'm not sure why this inconsistency is happening. Has anyone else experienced a similar issue, or could there be a delay in the message processing?

I encountered this too, but I think part of the issue is that when you get a 200 return code from sending a message, that isn't saying that the message has arrived, it's saying the request to send it has.
When I caught errors explicitly, I could see the underlying error message, usually a kick error like Too many special characters etc (helps to mod the bot). May have to change a few things but the below code had my bot sending the error it retrieved to chat, but you could optionally just log it to console.
 async sendMessage(channel, message) {     try {       const targetChannel = this.channels.find(ch => ch.data.chatroom.id === channel);       if (targetChannel) {         const response = await this.chat.api.chat.sendMessage(targetChannel.data.chatroom.id, message);         if (response.status && response.status.error) {           const { status, code, message: errorMessage } = response.status;           global.tools.logWithPlatform('kick',Extracted error details - Status: ${status}, Code: ${code}, Message: ${errorMessage});           throw new KickApiError('Error sending message', status, code, errorMessage);         }       }     } catch (error) {       //console.dir(error, { depth: null }); <- contents of error object       global.tools.logWithPlatform('kick', Error sending message: ${error.message});       const errorMessage = error.cause && error.cause.body && error.cause.body.status         ? Error: ${error.cause.body.status.message} (Code: ${error.cause.body.status.code}, Status: ${error.cause.body.status.status})        :Error: ${error.message};       await this.sendMessageToChannel(channel, errorMessage);     }   }

I have 10 bots sending messages to same chatroom and decided to put one second delay and didnt get any error so far. I dont know what changed but it seems fine for me. I will keep monitoring it to make sure it continues properly.


Update SortedMerge to use a PriorityQueue

Fixes #654

Looks like Test.Async.EndsWithTest.EndsWithWithStrings has some intermittent timing issue?

Thanks!


Bug: DOSee doesn't render correctly on macOS or iOS

The issue seems to be related to the pixel density used by the operating system vs the canvas resolution. There also might be a conflict between the Bootstrap 3 framework in use and hard coded values used by em-dosbox.
In CSS this hand be handled with though it will not work with all browsers.
https://developer.mozilla.org/en-US/docs/Web/CSS/@media/-webkit-device-pixel-ratio
@media
(-webkit-min-device-pixel-ratio: 2),
(min-resolution: 192dpi) {
    /* Retina-specific stuff here */
}

Test for CSS values by device: https://bjango.com/articles/min-device-pixel-ratio/
How to create cases in Chrome Devtools: https://developers.google.com/web/tools/chrome-devtools/device-mode/emulate-mobile-viewports

This has been moved to https://github.com/bengarrett/DOSee/issues/5


can not add azure text-embedding-ada-002 model

gpt35 added successfully.

i have successfully added it before, but can not add again after a fresh install...

Thank you for bringing this to our attention! We have taken note that Azure has changed the model name, and we will resolve this issue as soon as possible.


Header and sidebar broken on IE

This seems a general issue with CoreUI, the header and sidebar do not play nicely with Internet Explorer 10+ at small viewport sizes. The header drops the logo down and the navigation sidebar doesn't close appropriately with respect to the main body content.
Any ideas, has anyone fixed this? Works fine on Chrome.

Misplaced navbar-brand IE issue has been fixed with @coreui/coreui v2.0.10
Sidebar issue's not fixed yet

Thanks for the reply. xidedix solved it with a small amendment, a TOP I think iirc.
Now... onto that closing sidebar/non closing gap issue? ;)

working on it

@coreui/coreui v2.0.11 fixes sidebar gap issue at smaller viewport sizes

Fantastic, will check it out.


Access rights for repository shown as 'Read' despite writing is possible

Description
On the Settings -> Collaborators page of a repository the access rights for the team owning the repository is shown as 'Read'.
Members of the team can nevertheless push code and team repository access is set to 'Write' so I think it's just a visualization issue.
(Note: I cannot reproduce the issue on the demo site as I've got no right to create or modify teams there.)
Screenshots
Team access level shown as 'Read':

Team access level set to 'Write':

Gitea Version
1.18.3
Can you reproduce the bug on the Gitea demo site?
No
Operating System
Arch Linux
Browser Version
Firefox 109.0

I have the same issue.
Gitea Version
1.17.2
Can you reproduce the bug on the Gitea demo site?
No
Operating System
FreeBSD 13.1
Browser Version
Brave <IP_ADDRESS>, Firefox 108.0.2

Hi @lynix , right now the rule to compute the Access Mode is designed as: Return the minimal permission greater than "No Access" from the permission unit map except that all are "No Access".
In your case, i assumed that one or more of the permission selected from the unit map is selected to be "Read", and hence the Access Mode is computed as "Read".
If all units are selected to be "Write", then the Access Mode will be computed as "Write".

Thanks for the explanation, changing the access level for 'Releases' to 'No Access' made the collaborators summary show 'Write'.
That being said I'd like to add that I find this hidden computation very misleading. I'd wish to see something indicating that it is really not just 'Read', e.g. 'Mixed' with a mouse-over showing the differing unit permissions. Or add something indicating that it's the minimal access level being displayed.
Anyways, closing this as it turned out to be user confusion and not a bug.


Creating Zally issues on merge doesn't work

On each merge to master, we check if any content of the Guidelines has been changed. In case of a change, we create an issue in the Zally project.
This is done via bash script which is executed on Travis. Currently, it's failing with the following error message:
+create_zally_issue
++curl -s https://api.github.com/repos/zalando/restful-api-guidelines/commits/c7075d5bc17367d23cf8d21275889af22f6be0ab
++jq .commit.message
++sed 's/.* #\([0-9]*\)\+ .*/\1/'
+local pr_number=null
+changed_files=($(curl -s "${GH_REPO_URL}/pulls/${pr_number}/files" |         jq '.[] | .filename' | tr -d "\""))
++jq '.[] | .filename'
++tr -d '"'
++curl -s https://api.github.com/repos/zalando/restful-api-guidelines/pulls/null/files
jq: error (at <stdin>:4): Cannot index string with string "filename"
+local changed_files
+local content_changed=false
+'[' false = true ']'

Travis CI job

Fixed by #470, #471, #472, #473, #474.


Clarify the usage of siblings.

We should be consistent on what to expect in the sibling string field (Normalized value OR the original value).
More context
All sibling fields - _id  = normalized int enum , string = original value.
With the exception of OCSF siblings, activity, category, class, type.
Action Items -

A section explaining siblings in the readme
Update description to clarify the usage for all sibling fields

PR to update descriptions - https://github.com/ocsf/ocsf-schema/pull/272


Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$

use lib  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.BUILD-SNAPSHOT" % "provided",
or
"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta3" % "provided",
  val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicpMap).map(_._2).cache()
  val esregex = s"""(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}) (\\S+) (\\S+) \\[(.*)\\] "(.*?)" (\\d{3}) (\\S+) "(.*?)" "(.*?)" (.*?) (\\d+?)""".r

  val esRdd = lines.flatMap(_.split("\\n"))
              .map{
                case esregex(ip, client, user, dataTime, request, status, bytes, referer, agent, hit , origintime) => Map("ip" -> ip,"client"->client, "user" -> user, "dateTime" -> dataTime, "request" -> request, "status" -> status, "bytes" -> bytes, "referer"->referer, "agent" -> agent, "hit"-> hit, "origintime"-> origintime)
                case line => Map("message"-> line)
              }

esRdd.saveToEs("spark/docs")
or

sc.makeRDD(Seq(esRdd)).saveToEs("spark/docs")
or

EsSpark.saveToEs(esRdd, "spark/docs")
all has error ..

***Eerror:
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/package$
    at spark.c4hcdn.KafkaNetWork$.main(KafkaNetWork.scala:54)
    at spark.c4hcdn.KafkaNetWork.main(KafkaNetWork.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.package$
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 9 more

Was this issue really resolved ?

@charlesa101 It wasn't an issue to begin with - the reporter opened and close it right away suggesting it was a configuration issue. The CNFE typically occurs when the classpath is not set properly - in this case, if the es-hadoop/spark jar is not in the classpath.
If you encounter this/any issue, please open a new ticket with information about your environment.

You right @costin, i was trying to run a spark submit job
I specified the --jar path param after the --class and it was throwing exception due to that, works for me now! Thank you
[wrong]
~/spark-1.3.0-bin-hadoop2.4/bin/spark-submit  --class  --deploy-mode client   "hdfs://" --jars $(echo /path/to/jar/*.jar | tr ' ' ',')
[correct]
~/spark-1.3.0-bin-hadoop2.4/bin/spark-submit --jars $(echo /path/to/jar/*.jar | tr ' ' ',') --class  --deploy-mode client   "hdfs://"

Glad to hear it got sorted out. Anything we can do to prevent this type of
failure in the future?
Thanks,
On Wed, Apr 15, 2015 at 3:49 AM, charles adetiloye<EMAIL_ADDRESS>
wrote:

You right @costin https://github.com/costin, i was trying to run a
spark submit job
I specified the --jar path param after the --class and it was throwing
exception due to that, works for me now! Thank you
[wrong]
~/spark-1.3.0-bin-hadoop2.4/bin/spark-submit --class --deploy-mode client
"hdfs://" --jars $(echo /path/to/jar/*.jar | tr ' ' ',')
[correct]
~/spark-1.3.0-bin-hadoop2.4/bin/spark-submit --jars $(echo
/path/to/jar/*.jar | tr ' ' ',') --class --deploy-mode client "hdfs://"
—
Reply to this email directly or view it on GitHub
https://github.com/elastic/elasticsearch-hadoop/issues/297#issuecomment-93125975
.

Thanks @costin, I will try to update the documentation on this

can someone let me know how to solve above error cause ,i am facing same issue

@gaurav6351 As I've mentioned in #665, for questions and assistance please use the dedicated forum and not the issue tracker.
Ignoring this advice and commenting on issues closed 10 months ago is unlikely to result in any assistance, quite the opposite.
Further more, speaking about the issue at hand, the comment above indicates what was the problem and how it was solved.
Again, please pay a bit more attention to those trying to help you and consider following their advice. Thanks!

Another way is to use sbt assembly instead of sbt package to create your jar so that it includes all the required jars in the jar that is created.

I got it solved by using sbt assembly which includes all the required jars as @prabhushrikant mentioned


Feature: rewrite rules for relay and cache

It'd be most awesome if rewrite rules could be implemented in relay and cache, in the same way they are implemented in aggregator. This would allow for greater flexibility when designing metrics namespaces.

You can use carbon-aggregator-cache to run aggregation and cache in the same process.

@DanCech I find this section confusing.
Why does the carbon-aggregator-cache.py daemon need the relay-rules.conf file given that carbon-cache.py and carbon-aggregator.py do not? Does it also do relaying like carbon-relay.py does?
Also, which settings does the [aggregator-cache] section support in the carbon.conf file? It has no example section in the carbon.conf.example file. Maybe a union of both [cache] and [aggregator]?

That appears to be a typo, the relay-rules.conf would be used by carbon-aggregator, not carbon-aggregator-cache.
I haven't used it myself, but it appears that it will look for an [aggregator-cache] section that can contain all the settings applicable to the aggregator and the cache.


TLS 1.3 Key schedule: Second level secret generation

Context: This PR is part of the upstreaming of the TLS 1.3 prototype from https://github.com/hannestschofenig/mbedtls/, Specifically, it continues the upstreaming of the TLS 1.3 key schedule functionality, started in https://github.com/ARMmbed/mbedtls/pull/3568.
Details: Recall the TLS 1.3 key schedule form RFC 8448:

             0
             |
             v
   PSK ->  HKDF-Extract = Early Secret
             |
             +-----> Derive-Secret(., "ext binder" | "res binder", "")
             |                     = binder_key
             |
             +-----> Derive-Secret(., "c e traffic", ClientHello)
             |                     = client_early_traffic_secret
             |
             +-----> Derive-Secret(., "e exp master", ClientHello)
             |                     = early_exporter_master_secret
             v
       Derive-Secret(., "derived", "")
             |
             v
   (EC)DHE -> HKDF-Extract = Handshake Secret
             |
             +-----> Derive-Secret(., "c hs traffic",
             |                     ClientHello...ServerHello)
             |                     = client_handshake_traffic_secret
             |
             +-----> Derive-Secret(., "s hs traffic",
             |                     ClientHello...ServerHello)
             |                     = server_handshake_traffic_secret
             v
       Derive-Secret(., "derived", "")
             |
             v
   0 -> HKDF-Extract = Master Secret
             |
             +-----> Derive-Secret(., "c ap traffic",
             |                     ClientHello...server Finished)
             |                     = client_application_traffic_secret_0
             |
             +-----> Derive-Secret(., "s ap traffic",
             |                     ClientHello...server Finished)
             |                     = server_application_traffic_secret_0
             |
             +-----> Derive-Secret(., "exp master",
             |                     ClientHello...server Finished)
             |                     = exporter_master_secret
             |
             +-----> Derive-Secret(., "res master",
                                   ClientHello...client Finished)
                                   = resumption_master_secret

The vertical secret evolution is already upstreamed as mbedtls_ssl_tls1_3_evolve_secret(), and the primitive Derive-Secret() for the horizontal "second level" key evolution is upstreamed as mbedtls_ssl_tls1_3_derive_secret().
However, we have not yet upstreamed any code that would invoke mbedtls_ssl_tls1_3_derive_secret() with the specific TLS 1.3 labels, to generate the set of "second level" secrets specific to each handshake stage (early/handshake/application). This is what this PR does: It adds convenience structures
typedef struct
{
    unsigned char binder_key                  [ MBEDTLS_MD_MAX_SIZE ];
    unsigned char client_early_traffic_secret [ MBEDTLS_MD_MAX_SIZE ];
    unsigned char early_exporter_master_secret[ MBEDTLS_MD_MAX_SIZE ];
} mbedtls_ssl_tls1_3_early_secrets;

typedef struct
{
    unsigned char client_handshake_traffic_secret[ MBEDTLS_MD_MAX_SIZE ];
    unsigned char server_handshake_traffic_secret[ MBEDTLS_MD_MAX_SIZE ];
} mbedtls_ssl_tls1_3_handshake_secrets;

typedef struct
{
    unsigned char client_application_traffic_secret_N[ MBEDTLS_MD_MAX_SIZE ];
    unsigned char server_application_traffic_secret_N[ MBEDTLS_MD_MAX_SIZE ];
    unsigned char exporter_master_secret             [ MBEDTLS_MD_MAX_SIZE ];
    unsigned char resumption_master_secret           [ MBEDTLS_MD_MAX_SIZE ];
} mbedtls_ssl_tls1_3_application_secrets;

as well as helper functions
int mbedtls_ssl_tls1_3_derive_early_secrets(
          mbedtls_md_type_t md_type,
          unsigned char const *early_secret,
          unsigned char const *transcript, size_t transcript_len,
          mbedtls_ssl_tls1_3_early_secrets *derived );
int mbedtls_ssl_tls1_3_derive_handshake_secrets(
          mbedtls_md_type_t md_type,
          unsigned char const *handshake_secret,
          unsigned char const *transcript, size_t transcript_len,
          mbedtls_ssl_tls1_3_handshake_secrets *derived );
int mbedtls_ssl_tls1_3_derive_application_secrets(
          mbedtls_md_type_t md_type,
          unsigned char const *master_secret,
          unsigned char const *transcript, size_t transcript_len,
          mbedtls_ssl_tls1_3_application_secrets *derived );
int mbedtls_ssl_tls1_3_derive_resumption_master_secret(
          mbedtls_md_type_t md_type,
          unsigned char const *application_secret,
          unsigned char const *transcript, size_t transcript_len,
          mbedtls_ssl_tls1_3_application_secrets *derived );

implementing the second level key evolution.
It also upstreams a helper function
int mbedtls_ssl_tls1_3_create_psk_binder( mbedtls_ssl_context *ssl,
                               const mbedtls_md_type_t md_type,
                               unsigned char const *psk, size_t psk_len,
                               int psk_type,
                               unsigned char const *transcript,
                               unsigned char *result );

for the calculation of PSK binders.
Please see the corresponding commits, and esp. the documentation within mbedtls_ssl_tls1_3_keys.h, for more information.

@gabor-mezei-arm Thank you very much for your time and review. I have corrected the misleading documentation. As for the tests, I explained why I see no useful way of improving the tests at the moment, mainly due to a lack of public test vectors. Please let me know your thoughts.

@daverodgman @mpg @ronald-cron-arm Can this be merged?


Bash completions

I see that gron comes with Fish shell completions.
So I'd like to request Bash shell completions as well.

Hey @heitorPB and @tomnomnom ! I've added a PR that adds shell completion for Bash. #113


delete layered definitions

To handle layered dependencies, deleteElementTree must make multiple passes. In each pass, it creates and uses a new ElementTreeDeleter, which detects and classifies the remaining elements and calls IModelDb.deleteDefinitionElements on the remaining definition elements that it finds.
Note that deleteDefinitionElements will delete only definition elements that are not "used" (as defined by NativeDb usage calculator).
Scenario: A LineStyle refers to a GeometryPart (which captures a LS symbol). The GeometryStreams and SubCategories that refer to the LineStyle must be deleted first. Then the LS symbol GeometryPart can be deleted.
That's just one example. There can be further layers of dependencies.

Note that deleteDefinitionElements will delete only definition elements that are not "used" (as defined by NativeDb usage calculator).

This means that your attempt to delete the subtree may fail no matter how many passes you make, right?
You currently just loop maxPasses times (or until the top element is successfully deleted).
Do you have a way to detect that deletion will never succeed because of used definition elements, and bail before reaching maxPasses?


can't open file '/main.py' on container

Moving this here from the upstream funsearch repo.
          Hello,

I made somewhat simple implementation of the main missing components: https://github.com/jonppe/funsearch
* There's a 'Sandbox' implementation using a container with Podman or Docker

Can you add issues to your repository?
for example  if we "run the main Python process on a host computer outside of any container and let the process build and run separate sandbox containers (still requires Podman/Docker)."
the error is now in the sandbox:
/usr/local/bin/python3: can't open file '/main.py': [Errno 13] Permission denied

any help would be appreciated.
Originally posted by @Rubiel1 in https://github.com/google-deepmind/funsearch/issues/1#issuecomment-1902866554

@Rubie1 do you have Docker or Podman installed. Which version? Are you running Windows or Linux?
You could try setting the environment variable: LOGLEVEL=debug
and then re-running the command.
I'd check especially the command that looks something like this:
DEBUG:root:Executing: podman run --stop-timeout=30 -v /home/johannes/..../funsearch/funsearch/container/container_main.py:/main.py:ro -v /home/johannes/..../funsearch/data/1705907648/sandbox0/call1:/workspace -v /home/johannes/..../funsearch/data/1705907648/sandbox0/inputs/11.pickle:/input.pickle:ro funsearch_sandbox:latest /usr/local/bin/python3 /main.py /workspace/prog.pickle /input.pickle /workspace/output.pickle  2> data/1705907648/sandbox0/stderr_1.log
And then check also the the error file 'stderr_1.log' in this case.
But perhaps the log has other interesting parts too.
I haven't paid quite as much attention on Windows and Docker support so maybe there's something I missed.
The command tries to mount the container_main.py as /main.py inside the container but perhaps this doesn't work with some kind of folder structure or folder names in Windows...

Hello, I use Fedora 38 with 62.6 GiB of RAM.
I erased everything and installed funsearch again, and after podman  run... (in the workspace) I run
 llm install llm-gpt4all
which allowed me to specify the model
--model_name orca-mini-3b-gguf2-q4_0
After that the program runs, however,  after the 60 iteration it crashes. In fact I run it 3 times and each time it crashed after the 60 iteration.
The testing parameters of the last experiment were
functions_per_prompt: int = 2
  num_islands: int = 4
  reset_period: int = 4 * 60 * 60
  cluster_sampling_temperature_init: float = 0.1
  cluster_sampling_temperature_period: int = 30_000
  backup_period: int = 10
  backup_folder: str = './data/backups'

and
 programs_database: ProgramsDatabaseConfig = dataclasses.field(
      default_factory=ProgramsDatabaseConfig)
  num_samplers: int = 1
  num_evaluators: int = 1
  samples_per_prompt: int = 4

with messages
DEBUG:root:Executing: python /workspace/.venv/lib/python3.11/site-packages/funsearch/container/container_main.py /workspace/data/1705991385/sandbox0/call60/prog.pickle /workspace/data/1705991385/sandbox0/inputs/11.pickle /workspace/data/1705991385/sandbox0/call60/output.pickle  2> data/1705991385/sandbox0/stderr_60.log
DEBUG:root:Writing program to /workspace/data/1705991385/sandbox0/call60/program.py
Segmentation fault (core dumped)

Here is the stderr_60.log
sandbox0# cat stderr_60.log
Traceback (most recent call last):
  File "/workspace/.venv/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 59, in _wrapfunc
    return bound(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/.venv/lib/python3.11/site-packages/funsearch/container/container_main.py", line 27, in <module>
    main(sys.argv[1], sys.argv[2], sys.argv[3])
  File "/workspace/.venv/lib/python3.11/site-packages/funsearch/container/container_main.py", line 18, in main
    ret = func(input_data)
          ^^^^^^^^^^^^^^^^
  File "<ast>", line 17, in evaluate
  File "<ast>", line 37, in solve
  File "/workspace/.venv/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 1229, in argmax
    return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 68, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 45, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

I can reproduce the issue. This seems to be something related to the llm-gpt4all.
It seems to happen on the 61th repeated prompt even without funsearch involved e.g.

for i in range(70):
...   model.prompt("How are you?")
...
<Response prompt='How are you?' text=' As an AI, I don't have feelings or emotions like humans do. However, I am always ready and available to assist you in any way possible.'>
Segmentation fault (core dumped)

I couldn't immediately find any reported issues about it. I'll do a few more tests is this related to running inside container... perhaps some resource is running out.

Btw, I created the issue https://github.com/simonw/llm-gpt4all/issues/22 for the Segmentaion fault.
And in the latest funsearch version, improved the parsing to find the response properly from orca-mini-3b-gguf2-q4_0 responses. Anyway, at least with me the indentation in orca responses is usually 1 space.
Adding something like response.replace("\n ", "\n  ")  could be a workaround for orca but proper fix might require a bit more work.


Jasmine test crashes Karma in Firefox before completion of run

This test seems to run ok in Jasmine standalone but crashes my Karma setup:
it('should catch error thrown by JSON.parse', function (done) {
  new Promise(function (resolve) {
    resolve('notjson');
  }).then(JSON.parse)
  .catch(function () {
    console.log('caught');
    expect(true).toBe(true);
    done();
  });
});

Karma config:
var TEST_FILES = 'test/integration/promiseSpec.js';

module.exports = function(config) {
  config.set({

    files: [
      TEST_FILES
    ],

    frameworks: ['jasmine'],

    browsers: ['Firefox'],

    reporters: ['mocha'],

  });
};

It runs fine with Chrome. I am running Karma 0.12.37

Can you give the full stack trace of the errors please?

There is no stack trace, only SyntaxError: JSON.parse: unexpected keyword at line 1 column 1 of the JSON data

After experimenting a bit more, wrapping JSON.parse in another function produces the expected result of the test passing and test execution continuing:
var jsonparse = function (toParse) { return JSON.parse(toParse); };

it('should catch error thrown by JSON.parse', function (done) {
  new Promise(function (resolve) {
    resolve('notjson');
  }).then(jsonparse)
  .catch(function () {
    console.log('caught');
    expect(true).toBe(true);
    done();
  });
});

Interesting, what Promise library are you using?

I'm currently using Firefox's native ES6 promises

It looks like that is a bug in Firefox then, because executing this on its own in the Firefox console, I get the same result as you do.

Swapping out Firefox's native implementation for core-js ES6 promises (via Babel) makes the tests run as expected so perhaps there is some issue with the way Firefox is propagating this particular exception?

Yes pretty sure it's a bug in their promise implementation.

Have created a bug report here: https://bugzilla.mozilla.org/show_bug.cgi?id=1182758

Closing, as there is nothing we can do from this side.


Support custom actions to be executed on a document type

Allow a document definition to define fully custom action functions to be executed when the sync function is run. It should allow at least two different types:

custom functions that are run after the document's type is identified, but before user authorization is verified (e.g. customPreAction)
custom functions that are run after user authorization is verified and the document's contents pass validation, but before channels are assigned (e.g. customPostAction)

For example (not to be considered final):
myDocType: {
  channels: ...,
  typeFilter: ...,
  propertyValidators: ...,
  customPreAction: function(doc, oldDoc) {
    // Do something fancy before any validation is performed
  },
  customPostAction: function(doc, oldDoc) {
    // Do something fancy after all validation has passed
  }
}

The implementation in pull request #62 implements custom action support for the following events:

When a document's type is identified
When authorization is granted to the user
When document contents are validated
When access is assigned to users/roles
When channels are assigned to the document

Addressed by #62 and #63


Angular 2.0.0 Final: core_1.provide is not a function

Getting stuck on the following error using the Angular2 Final version:
Error: TypeError: core_1.provide is not a function
at Object.eval (http://localhost:5555/node_modules/angular2-adal/core.js:15:12)
at eval (http://localhost:5555/node_modules/angular2-adal/core.js:20:4)
at eval (http://localhost:5555/node_modules/angular2-adal/core.js:21:3)
at Object.eval (http://localhost:5555/app/shared/trip/trip.service.js:15:14)
Evaluating http://localhost:5555/node_modules/angular2-adal/core.js
Evaluating http://localhost:5555/app/shared/trip/trip.service.js
Evaluating http://localhost:5555/app/shared/trip/index.js
Evaluating http://localhost:5555/app/shared/index.js
Evaluating http://localhost:5555/app/app.component.js
Evaluating http://localhost:5555/app/app.module.js
Evaluating http://localhost:5555/app/main.js
Error loading http://localhost:5555/app/main.js "Report this error at https://github.com/mgechev/angular2-seed/issues"(anonymous function) @ (index):110ZoneDelegate.invoke @ zone.js?1474011477542:192Zone.run @ zone.js?1474011477542:85(anonymous function) @ zone.js?1474011477542:451ZoneDelegate.invokeTask @ zone.js?1474011477542:225Zone.runTask @ zone.js?1474011477542:125drainMicroTaskQueue @ zone.js?1474011477542:357ZoneTask.invoke @ zone.js?1474011477542:297
Anyone got any suggestions?

I'm having the same issue. There was a change in angular to how dependency injection is handled that may be causing this but I haven't been able to track it down yet in the time I've had. It's possibly in one of the libraries this depends on.

I am having the same problem.

I "fixed" this by simply copying the adal.service.ts and oauthdata.model.ts into my own project rather than fetching them with NPM. You will have to get the "adal-angular" npm module and save it into your project, and also add the necessary "adal" @types or typings to your project.
Personally I use the angular2-seed project as a base for my solution, and I also added these lines to my seed.config.ts:
SYSTEM_CONFIG_DEV: any = {
...
packageConfigPaths: [
/node_modules/adal-angular/package.json,
...
paths: {
'adal/': 'node_modules/adal-angular/',
...
packages: {
'adal': {
main: 'adal.js',
defaultExtension: 'js'
}
},
map: {
'adal': ${this.APP_BASE}node_modules/adal-angular/lib/adal
}
...
SYSTEM_BUILDER_CONFIG: any = {
...
packageConfigPaths: [
join(this.PROJECT_ROOT, 'node_modules', 'adal-angular', '*', 'package.json')
...
paths: {
'adal': {
main: 'lib/adal.js',
defaultExtension: 'js'
}
},
map: {
'adal': join(this.PROJECT_ROOT, 'node_modules/adal-angular')
}
and in my typings.json:
"adal": "github:DefinitelyTyped/DefinitelyTyped/adal-angular/adal.d.ts#53347571e6f44ed9a961b569bb81893fde4789c1",
I can now use the the service in all my components:
import { AdalService } from '../adal/index'; (/index because i made my own index.ts which exports the adal service)
And in my main app component:
constructor(private adalService: AdalService)
adalService.init({
tenant: 'guid',
clientId: 'guid',
endpoints: {'https://websiteurl': 'https://company.onmicrosoft.com/app'},
cacheLocation: 'localStorage'
});
adalService.handleWindowCallback();
This allows you to make a very simple AuthGuard:
import { Injectable } from '@angular/core';
import { CanActivate, Router } from '@angular/router';
import { AdalService } from '../adal/index';
@Injectable()
export class AuthGuard implements CanActivate {
constructor(private adalService: AdalService, private router: Router) {}
canActivate() {
if (!this.adalService.userInfo.isAuthenticated) {
this.adalService.login();
return false;
} else {
console.log('AuthGuard: User is logged in as ' + this.adalService.userInfo.userName);
return true;
}
}
}
Hope this helps anyone. It's not solution yet for the problem in this library, but for me it's a workaround until this library (or Microsoft comes with their own Angular2 module) is updated for implementation with Angular2 Final.

If you use the below construct in core.ts for exporting one of your errors should go away, but I haven't been able to go beyond to fully make use of the library.
export const ANGULAR2_ADAL_PROVIDERS: any[] = [
{
provide: AdalService,
useClass: AdalService
}
];
instead of
provide(AdalService, {useClass: AdalService})

+1 See also: https://github.com/sureshchahal/angular2-adal/issues/3

This is definitely fixed by just recreating the service itself within a project, but I've been looking to figure out what the issue is as well. It's a side-project at this point, but I'm working on it.
It's not necessarily something in the libraries this depends on since it works fine, just apparently not as a separate module for some reason.

Anyone have luck using webpack 2 to load the package? I also need to use it in an Ionic 2 project which does not have Angular Router.

This is now fixed in the ng2-adal, see: https://github.com/sureshchahal/angular2-adal/issues/3

We are also exposing it via NgModules now!!
https://github.com/sureshchahal/angular2-adal/issues/8

I too had this issue. Here's an example where it's completely fixed. This repo is based on Angular2-Quickstart: https://github.com/ranveeraggarwal/ng2-adal-QuickStart


Db_path/config unbound

The variable db_path is not bound in the case that the if statement above is not being executed. Same for the config variable

https://github.com/Porthmeus/MeMoMe/blob/669b67cefe7ccfc1236a238d46d47f3f62418dfb/src/annotateChEBI.py#L20

Is removed the if statement on my branch, will through a correct error next time. Will be merged with next PR.

Fixed with #90


External Authentication is not working in Combined AspnetEfCore Example

The external Google authentication (or Facebook) is not working in the quickstart or any other "new" AspCore 2 template I tried.
The method SignInManager.GetExternalLoginInfoAsync() always returns null. I already tried the following variations in the Startup Config.
 services.AddAuthentication()
                .AddFacebook(options =>
                {
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

 services.AddAuthentication()
                .AddFacebook("Facebook", options =>
                {
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

 services.AddAuthentication()
                .AddFacebook("Facebook", options =>
                {
                    options.SignInScheme = IdentityConstants.ExternalScheme;
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

 services.AddAuthentication()
                .AddFacebook(options =>
                {
                    options.SignInScheme = IdentityConstants.ExternalScheme;
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

 services.AddAuthentication()
                .AddFacebook("Facebook", options =>
                {
                    options.SignInScheme = IdentityServerConstants.ExternalCookieAuthenticationScheme;
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

 services.AddAuthentication()
                .AddFacebook(options =>
                {
                    options.SignInScheme = IdentityServerConstants.ExternalCookieAuthenticationScheme;
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];
                });

None of those Options work. My current full Startup.cs class looks like this:
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddDbContext<ApplicationDbContext>(options =>
                options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));

            services.AddIdentity<ApplicationUser, IdentityRole>()
                .AddEntityFrameworkStores<ApplicationDbContext>()
                .AddDefaultTokenProviders();

            // Add application services.
            services.AddTransient<IEmailSender, EmailSender>();

            services.AddMvc();

            // configure identity server with in-memory stores, keys, clients and scopes
            services.AddIdentityServer()
                .AddDeveloperSigningCredential()
                .AddInMemoryPersistedGrants()
                .AddInMemoryIdentityResources(Config.GetIdentityResources())
                .AddInMemoryApiResources(Config.GetApiResources(Configuration))
                .AddInMemoryClients(Config.GetClients(Configuration))
                .AddAspNetIdentity<ApplicationUser>();

            services.AddAuthentication()
                .AddFacebook(options =>
                {
                    options.ClientId = Configuration["ExternalLoginProviders:Facebook:ClientId"];
                    options.ClientSecret = Configuration["ExternalLoginProviders:Facebook:ClientSecret"];

                    List<string> fields = new List<string>();
                    Configuration.GetSection("ExternalLoginProviders:Facebook:Fields").Bind(fields);
                    foreach (string field in fields)
                    {
                        options.Fields.Add(field);
                    }

                    List<string> scopes = new List<string>();
                    Configuration.GetSection("ExternalLoginProviders:Facebook:Scopes").Bind(scopes);
                    foreach (string scope in scopes)
                    {
                        options.Scope.Add(scope);
                    }
                });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
                app.UseBrowserLink();
                app.UseDatabaseErrorPage();
            }
            else
            {
                app.UseExceptionHandler("/Home/Error");
            }

            app.UseStaticFiles();

            app.UseIdentityServer();

            MigrateAndSeedDatabase(app);

            app.UseMvc();
        }

        public virtual void MigrateAndSeedDatabase(IApplicationBuilder app)
        {
            using (IServiceScope serviceScope = app.ApplicationServices.GetRequiredService<IServiceScopeFactory>().CreateScope())
            {
                serviceScope.ServiceProvider.GetService<ApplicationDbContext>().Database.Migrate();
            }
        }
    }

Any ideas why the external authentication does not work?

This seems to be a general question about IdentityServer - not a bug report or an issue.
Please use one of the our free or commercial support options
See here for more details.
Thanks!

This is a Bug since you can reproduce this issue by simply cloning the
example repo.
External authentication in the combined quickstart does not work.
On Sun 12. Nov 2017 at 16:03, Dominick Baier<EMAIL_ADDRESS>wrote:

This seems to be a general question about IdentityServer - not a bug
report or an issue.
Please use one of the our free or commercial support options
See here
https://identityserver4.readthedocs.io/en/release/intro/support.html
for more details.
Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IdentityServer/IdentityServer4.Samples/issues/240#issuecomment-343743471,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuGf4GW1cLHtCBC486CvXc2-lvGqXRBks5s1wjdgaJpZM4Qa6MV
.

you said

or any other "new" AspCore 2 template I tried.

This sounds to me that you have the problem also "outside" of IdentityServer..?!

No with that I meant the other quickstart Templates that have been upgraded
to asp core 2.
On Sun 12. Nov 2017 at 16:40, Dominick Baier<EMAIL_ADDRESS>wrote:

you said
or any other "new" AspCore 2 template I tried.
This sounds to me that you have the problem also "outside" of
IdentityServer..?!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IdentityServer/IdentityServer4.Samples/issues/240#issuecomment-343745717,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuGf9aaFnZNVFlu7dmAg9BTt988MQ_5ks5s1xFhgaJpZM4Qa6MV
.


Subnet previously used by deleted api management service cannot be deleted

Steps to reproduce

Create a apim service
{"location":"westeurope","properties":{"certificates":[],"customProperties":{"Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls11":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Ciphers.TripleDes168":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11":"false"},"hostnameConfigurations":[],"publisherEmail":"pub1@email.com","publisherName":"pub1","virtualNetworkType":"None"},"sku":{"name":"Developer","capacity":1},"tags":{}}
2020/08/10 11:59:51 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions/.../resourceGroups/acctestRG-200810115925124829/providers/Microsoft.ApiManagement/service/acctestAM-200810115925124829?api-version=2019-12-01:
HTTP/2.0 201 Created

Update the apim service with VN Subnet, change to Internal virtualNetworkType
PUT /subscriptions/.../resourceGroups/acctestRG-200810120050691536/providers/Microsoft.ApiManagement/service/acctestAM-200810120050691536?api-version=2019-12-01 HTTP/1.1
{"location":"westeurope","properties":{"certificates":[],"customProperties":{"Microsoft.WindowsAzure.ApiManagement.Gateway.Protocols.Server.Http2":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls11":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Ciphers.TripleDes168":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11":"false"},"hostnameConfigurations":[],"notificationSenderEmail":"apimgmt-noreply@mail.windowsazure.com","publisherEmail":"pub1@email.com","publisherName":"pub1","virtualNetworkConfiguration":{"subnetResourceId":"/subscriptions/.../resourceGroups/acctestRG-200810120050691536/providers/Microsoft.Network/virtualNetworks/acctestVNET-200810120050691536/subnets/acctestSNET-200810120050691536"},**"virtualNetworkType":"Internal"**},"sku":{"name":"Developer","capacity":1},"tags":{}}

Update the apim service, remove Subnet setting, change to None virtualNetworkType
PUT /subscriptions/.../resourceGroups/acctestRG-200810120050691536/providers/Microsoft.ApiManagement/service/acctestAM-200810120050691536?api-version=2019-12-01 HTTP/1.1
{"location":"westeurope","properties":{"certificates":[],"customProperties":{"Microsoft.WindowsAzure.ApiManagement.Gateway.Protocols.Server.Http2":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls11":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Ciphers.TripleDes168":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Ssl30":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10":"false","Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11":"false"},"hostnameConfigurations":[],"notificationSenderEmail":"apimgmt-noreply@mail.windowsazure.com","publisherEmail":"pub1@email.com","publisherName":"pub1","**virtualNetworkType":"None"**},"sku":{"name":"Developer","capacity":1},"tags":{}}

Delete them all, all has been deleted, except the VN, SN and the resource group. And the SN deletion returns error:  StatusCode=400 -- Original Error: Code="InUseSubnetCannotBeDeleted" Message="Subnet acctestSNET-200810094556761168 is in use by /subscriptions/…/providers/Microsoft.ApiManagement/service?vnetResourceGuid=d1f6670d-d8a4-48e9-9226-94ae9e5dd63e&subnet=acctestSNET-200810094556761168&api-version=2016-07-07 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet." Details=[]
2020/08/10 13:52:06 [DEBUG] AzureRM Request:
DELETE /subscriptions/.../resourceGroups/acctestRG-200810120050691536/providers/Microsoft.ApiManagement/service/acctestAM-200810120050691536?api-version=2019-12-01 HTTP/1.1
{
"status": "Succeeded"
}

2020/08/10 13:52:08 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions/.../providers/Microsoft.Network/locations/westeurope/operations/b8d84514-9520-4617-9ad3-ea2327d6ac7b?api-version=2020-05-01:
DELETE /subscriptions/.../resourceGroups/acctestRG-200810120050691536/providers/Microsoft.Network/virtualNetworks/acctestVNET-200810120050691536/subnets/acctestSNET-200810120050691536?api-version=2020-05-01 HTTP/1.1
{
"error": {
"code": "InUseSubnetCannotBeDeleted",
"message": "Subnet acctestSNET-200810120050691536 is in use by /subscriptions/.../providers/Microsoft.ApiManagement/service?vnetResourceGuid=a5bc5d2b-a9b3-477b-90bf-7dc2146149b8&subnet=acctestSNET-200810120050691536&api-version=2016-07-07 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
"details": []
}
}
5.	On portal, we could not delete the SN either

The scenario where we move the service out of Virtual Network does not remove the link to subnet immediately.
This is to avoid downtime for customer Apps, as the DNS refresh can take some time. When the service is moved out of the Virtual network, the IP Address changes. For more details refer here
On Update, the link is removed within 3 hours. In this case, the Link was removed at 2020-08-10 08:01 UTC. The service was moved out of VNET at 2020-08-10 05:50 UTC.
This is BY DESIGN!


[SRE-122] Fix docker container builds

gitrev was redefined in commit 70674b3 which broke docker container builds.
This PR continues that change down into dockerImage.

bors r+


Fix typo in README.md

fix duplicate readme line from yarn to npm

Why are we moving to npm?

Examples include yarn and npm above. Updating this bit to not have duplicate yarn and yarn

You can now do stuff!

yarn hardhat test

or
yarn hardhat test

LGTM. Will just get Andrej to verify, as i believe he owns this starter kit now


Support VFIO backed VFs

When SR-IOV device plugin allocates a VFIO typed VF, the device doesn't have a netlink representation. So when SR-IOV CNI then tries to move the device into the pod, it fails because netlink interface is not present.
While one may argue that in case of VFIO devices we may not need a CNI at all, Multus requires that a network has type specified. If it's not sriov then it should be something else. We could write a new CNI plugin that would do nothing in terms of device binding into the namespace; but it's not convenient if we already have SR-IOV CNI with meaningful type name. We could also benefit from SR-IOV CNI handling the binding if we e.g. want IPAM info to be allocated for the device (perhaps because later we will use this information to pass it into software running inside the pod; in case of kubevirt it would be libvirt-backed VM that receives IPAM configuration via cloud-init; https://github.com/intel/sriov-cni/issues/37 is the place where one of potential approaches to pass IPAM into pod is discussed.)
It's my belief that SR-IOV CNI should gracefully handle VFs that are registered with VFIO driver like it already does for non-VFIO devices. This issue is just that - making the plugin not fail on VFIO registered VF and return desired IPAM information in result returned to the caller.

@zshi-redhat @ahalim-intel what's the official strategy for userspace devices for this CNI plugin? Would a PR that would make binding for e.g. VFIO devices be accepted into the CNI plugin, or should we go and contribute to another repo? (Perhaps start a new one.)
I'd like to make sure that we are on the same page before I start working on the implementation.
The implementation would resemble the shell CNI plugin mentioned in https://github.com/intel/sriov-cni/issues/37#issuecomment-463841511 but as part of the SR-IOV CNI plugin.

/assign booxter

@zshi-redhat @ahalim-intel what's the official strategy for userspace devices for this CNI plugin?

There is a repo for userspace CNI plugins, it supports userspace interfaces with OVS or VPP as backend.

Would a PR that would make binding for e.g. VFIO devices be accepted into the CNI plugin, or should we go and contribute to another repo? (Perhaps start a new one.)

Could you please elaborate a little bit on how SR-IOV CNI knows which driver the VF shall be bind to?

I'd like to make sure that we are on the same page before I start working on the implementation.
The implementation would resemble the shell CNI plugin mentioned in #37 (comment) but as part of the SR-IOV CNI plugin.

@zshi-redhat @ahalim-intel what's the official strategy for userspace devices for this CNI plugin?

There is a repo for userspace CNI plugins, it supports userspace interfaces with OVS or VPP as backend.

Would a PR that would make binding for e.g. VFIO devices be accepted into the CNI plugin, or should we go and contribute to another repo? (Perhaps start a new one.)

Could you please elaborate a little bit on how SR-IOV CNI knows which driver the VF shall be bind to?

I'd like to make sure that we are on the same page before I start working on the implementation.
The implementation would resemble the shell CNI plugin mentioned in #37 (comment) but as part of the SR-IOV CNI plugin.

Would it make sense that we do the binding of VF in SR-IOV CNI by providing a net-attach-def Custom Resource with dpdk-driver as one of its parameters?  Then we could have SR-IOV CNI check whether additional driver(dpdk-driver) is configured and do the binding accordingly. This doesn't solve the problem of publishing IPAM information to container Pod, but it avoid device plugin to track all the vfio device and
avoid having another shell CNI to do the binding.

@zshi-redhat the device cgroup permissions for the container cannot be taken care of with a CNI plugin. Thats why it makes more sense to leave driver binding/unbinding out of a CNI plugin.  We had this in sriov-cni as a legacy feature(before any notion of device plugin). A device plugin is right place for device file permission related issue. The SRIOV DP takes care of it already, thus we no longer should do anything with driver changes in the CNI plugin.

@ahalim-intel @zshi-redhat I think we can determine the current driver for a device using sysfs. If it's VFIO, do nothing to move (non-existing) netlink interface into namespace. (But still we can create a veth to carry IPAM information. That would be a separate follow-up feature.)

@zshi-redhat

Would it make sense that we do the binding of VF in SR-IOV CNI by providing a net-attach-def Custom Resource with dpdk-driver as one of its parameters?

Note that in my use case, it's not DPDK at all, it's a regular VF that happens to be registered with VFIO userspace driver. So using dpdk_driver for this need seems incorrect. I also think that it should be easy to automatically detect whether a device is bound to vfio driver and act accordingly (by skipping the move-netlink-to-namespace step of the plugin).


Base manager implementation

@bilfeldt no option for requesting review so tagging here

@Hasnayeen I don't think there is any reason for the manager to implement the interface of the driver.


Unit Converter

Describe the solution you'd like
Using the unit-converter PyPi module I would like to be able to work on a unit converter. This converter would run with the command !convert.  This command would allow for unit conversion such as celsius to Fahrenheit and also conversion between metric and US customary units.
Command:
!convert <value> <units> to <units>
Examples:

!Convert 2.78 daN*mm^2 to mN*µm^2

!Convert 10 min to h

!Convert 10 ℃ to ℉

I would like to be able to work on this.

Looks like a good idea to me!


Add range submodule

This introduce a new 'range' module
to support encoding, decoding and validation of HTTP range requests.
This could be a helper to implement http-rs/tide#63 and enhance tide's serve_dir with range requests.
This is also an experiment toward typed headers discussed in #99.
I am using a simple design where the typed header implements TryFrom and ToHeaderValues. This has the advantage to be fully compatible with the current header APIs. However I believe we could push Rust type system a little further.
For instance we could enhance the Headers type with a new method and trait:
trait TypedHeader: ToHeaderValues {
    const HeaderName: &'static str;
}

impl Headers {
    // TODO: find a better name
    pub fn get_typed<T>(&self) -> Option<T>
    where
            T: TypedHeader,
    {
        // TODO: implementation
    }
}

One could then retrieve a typed value directly from the headers :
let range: Option<Range> = headers.get_typed();

I am not sure if it works, but I could add a few commits on this pull request if you want a try.
Thanks in advance for your comments !
Cheers.

We should probably do it in a separate PR, and I'm not sure how idiomatic this would be, but Headers::get::<TypedHeader>() would seem reasonable to me.

We should probably do it in a separate PR, and I'm not sure how idiomatic this would be, but Headers::get::<TypedHeader>() would seem reasonable to me.

Indeed this is orthogonal to this PR and deserve a separate discussion.

Hello @yoshuawuyts,
In the scope of #99 I see you are progressing well on #222, #224 , #225, #220.
What do you expect from me at this point: You would be happy that I rework this pull request once the above PRs are merged and reuse the same conditional APIs ? Or you plan to implement this yourself ?
Some part of this PR need to be reworked, I identified a few bugs (regarding RFCs), I prefer to wait for your comments before going forward.
Cheers,

@ririsoft it's been a few weeks since we've actually released the patches you linked above, and we're definitely on track in that direction. Apologies that it took a while to figure out how we wanted the headers structured. If you're still interested in adding typed headers for Range that'd be a very welcome contribution! It'd be so good to have https://github.com/http-rs/tide/pull/687 too!

@ririsoft it's been a few weeks since we've actually released the patches you linked above, and we're definitely on track in that direction. Apologies that it took a while to figure out how we wanted the headers structured. If you're still interested in adding typed headers for Range that'd be a very welcome contribution! It'd be so good to have http-rs/tide#687 too!

Sure, I followed the discussions and will rewrite this PR with latest design. I am just a bit busy at work those days, I plan to come back on this next week.
Cheers.

I have resumed the work here: https://github.com/ririsoft/http-types/tree/headers-range-wip
and will squash/rebase/merge back to this PR once ready for review (a couple of days).

Sorry guys, I worked less on this than initially expected but I am progressing and almost there.

Hi @yoshuawuyts,
The work is ready for review on my side.
You might want to flag this as unstable so that we can experiment the API with Tide before freezing it.
I am very open to any suggestions of course. Also the RFC specification is not straightforward, you might want to have a 4 eyes check on my implementation.
Cheers.

Hi @yoshuawuyts thank you very much for your review. I am pretty much in line globally.

Assume the RangeUnit is always bytes?
The RFC states that RangeUnits can be expected to be expanded in the future through RFC. Though in the past 6 years this has not happened, which makes it questionable to which degree we need to consider this. We could make the assumption that the only two types that may ever be sent are "none" and bytes.

I initially worked on a version with a RangeUnit enum, like the one you mentioned, opening the door to other range units if the RFC evolves in the future. But I realized that this is a lot of boilerplate for every day use since only bytes unite are used today in practice. So I went for a compromise with this Byte prefix thing which is not very much satisfying at the end, I agree.

Instead I would like to suggest switching it around: create a RangeUnits enum which is tagged #[non_exhaustive] and None and Bytes members, and then have Range , ContentRange, and AcceptRanges headers. This will allow us to add more range types in the future without breaking anything.

I would go for that design too, but I am not sure about having None as an enum variant: None is not a range unit type. It is a possible value for the AcceptRanges header to notify that range requests are not accepted, but it is not used otherwise for Range and ContentRange headers. We could have AcceptRanges::from_headers return Ok(None) if the Accept-Range header has the none value, without any impact client side.
However, by removing None from the enum we would have only one variant which looks strange too.
A third option would be something like:
enum RangeUnit {
    Bytes
    Other(String)
}

This allows for usage of non standard units, but not great either.
If we go for the RangeUnit design we will need to handle it with the Range and ContentRange headers too. Those type do not only returns a range unit but also a range set (ByteRanges in my current implementation) and a single Range. At the end I do not see how we can avoid several if let and match  code for the end user using only bytes unit in practice. Do you have something in mind that could help here ?

suffix-length
I was reading about ranges on mdn and I'm confused about how the suffixes are exposed in our API? Do you have a good sense of how they work, and how we could implement them?

You can see them as relative seek from either start or end. So far I do not provide further helper over it, I am just exposing them with an optional value. In Tide we will have to match over them to either SeekFrom::Start or SeekFrom::End. We can extend the type with further helper functions if need be without a breaking change I believe.

Conclusion
Despite having feedback on some of the design, I think we're very well on our way to getting where we need to be! — I'm incredibly excited for this work, and how it'll enable us to implement http-rs/tide#687. Thanks so much for your work on this!

Thank you for your great feedback. This leaves us with the following choices:

We provide implementation for bytes unit only, and we fully assume a breaking change will probably need to be introduced if a new unit is added to the RFC. This is a limited risk at the benefit of simple code covering 100% of usage for the last 6 years.
We go for a RangeUnit enum #[non_exhaustive], at the benefit of allowing a source compatible introduction of a new unit, but at the risk to "complexify" the API for every day use.

Not an easy decision.
Do you want me to have a try with the RangeUnit enum design to see how it goes, or you want to listen to the little angel ?
In the latter case I will rename ByteRanges to Range and ByteRange to RangeItem (or any better name you might have), and simplify the API to handle bytes only.
What do you think ?
After this we need to talk about this comment which could impact the design of the If-Range header. But one discussion at a time :)

Assume the RangeUnit is always bytes?

However, by removing None from the enum we would have only one variant which looks strange too.

I think that's fine though, especially if we mark the enum as #[non_exhaustive] and maybe add a comment this may be extended in the future.

At the end I do not see how we can avoid several if let and match code for the end user using only bytes unit in practice. Do you have something in mind that could help here ?

Not necessarily; I think at this layer it's more important that the implementation is correct and understandable, than necessarily providing the best end-user ergonomics. Then in Tide we can use the implementation we've defined here to truly make it ergonomic.
suffix-length

We can extend the type with further helper functions if need be without a breaking change I believe.

Fantastic! Happy to punt this topic for now then (:
Conclusion

Do you want me to have a try with the RangeUnit enum design to see how it goes?

I think this is indeed the route I'd like to see explored. Thanks so much for taking the time to work through this; I feel I say this everytime it comes up, but I'm incredibly excited for this!

I think this is indeed the route I'd like to see explored.

Here we are ! The new design is  enum  based but different than what we discussed above. I would like to apologies for that and hope you won't be too disappointed.
I came to this design because each header exposes different types that I could not unify across one single Unit enum.
In this design each header is an enum, all with a Bytes variant but they all differ in the inner types exposed. All bytes related implementation belongs to a dedicated bytes module.
There is a few points that I was not sure about, such having new constructors taking a impl Into<...> for each header.
Do not hesitate to challenge my proposal I am fully open for suggestions.

@ririsoft In https://github.com/http-rs/http-types/pull/180#pullrequestreview-540126702 I proposed to go for RangeUnits + RangeDirective types so that one can hold the unit definition, and the other can work as the direct inputs to the headers.
In https://github.com/http-rs/http-types/pull/180#issuecomment-740007234 it sounds like you attempted to unify these into a single type, which then didn't work out -- am I reading that correctly?

@ririsoft In #180 (review) I proposed to go for RangeUnits + RangeDirective types so that one can hold the unit definition, and the other can work as the direct inputs to the headers.
In #180 (comment) it sounds like you attempted to unify these into a single type, which then didn't work out -- am I reading that correctly?

Hello @yoshuawuyts ,
I do not understand. Were you able to review my latest updates with enum ?
Does it matches your expectations ? Otherwise would you be able to provide some code snippets for one of the header type please. I really don't see where you want to go, I am really sorry about that.

@ririsoft that's all good! -- I think what I may do is fork your PR, and then file a patch against your PR. I think perhaps at this point that may be the easiest way to communicate what I mean 😅

@ririsoft that's all good! -- I think what I may do is fork your PR, and then file a patch against your PR. I think perhaps at this point that may be the easiest way to communicate what I mean 😅

@ririsoft that's all good! -- I think what I may do is fork your PR, and then file a patch against your PR. I think perhaps at this point that may be the easiest way to communicate what I mean 😅

Yeah we are reaching the limit of asynchronous talk here I am afraid :). A small patch says a lot more sometimes 👍
Also I had a look at what they did on hyperium: https://docs.rs/headers/0.3.3/headers/struct.Range.html.
Interestingly they are using the RangeBounds type from std. It may look like a good idea but I am not sure how you can represent a "negative bound" such as "the last five bytes" that you can have with http range requests such -5. To be further discussed with your patch ...

@ririsoft that's all good! -- I think what I may do is fork your PR, and then file a patch against your PR. I think perhaps at this point that may be the easiest way to communicate what I mean 😅

Yeah we are reaching the limit of asynchronous talk here I am afraid :). A small patch says a lot more sometimes 👍
Also I had a look at what they did on hyperium: https://docs.rs/headers/0.3.3/headers/struct.Range.html.
Interestingly they are using the RangeBounds type from std. It may look like a good idea but I am not sure how you can represent a "negative bound" such as "the last five bytes" that you can have with http range requests such -5. To be further discussed with your patch ...

I am not using http-rs anymore and I am thus not interested in working on this PR further.


#11345 Boolean text presentation configurator

Please note that this PR does not include:

Image checkbox colorization support
Assertions against malicious input in text mode

To do list:

[x] ~Rewrite configuration page code because event system is ugly~
[x] ~Checkboxes in property tree viewer are aligned to the left while resting ignoring the user-defined configuration~
[x] ~Verify user input before saving (avoid having empty values)~
[x] ~Localize strings~

Any custom text input available?

No bold text option?

Current implementation does not have such option. See to-do note: https://github.com/dbeaver/dbeaver/issues/11345#issuecomment-920173213

Current implementation does not have such option. See to-do note: #11345 (comment)

Got it. Deleted my comment here as soon as I saw your edited comment there. Asychronous user-developer interaction :) But relink in issue 11345 still there. A bug of Github(?)

You mean the mention? It will persist because that's how GitHub works :^)


Roll Engine from 54e1f3f4bb36 to 03ea5aa51a4a (34 revisions)

Roll Engine from 54e1f3f4bb36 to 03ea5aa51a4a (34 revisions)
https://github.com/flutter/engine/compare/54e1f3f4bb36...03ea5aa51a4a
2022-01-26<EMAIL_ADDRESS>Roll Skia from cec7f5dec541 to a54564efd464 (2 revisions) (flutter/engine#31087)
2022-01-26<EMAIL_ADDRESS>Roll Dart SDK from 10875d49378b to 407ce0cb852b (1 revision) (flutter/engine#31086)
2022-01-26<EMAIL_ADDRESS>Roll Skia from 1d9471f5531a to cec7f5dec541 (2 revisions) (flutter/engine#31085)
2022-01-26<EMAIL_ADDRESS>Roll Skia from 2b80cb6636ef to 1d9471f5531a (1 revision) (flutter/engine#31083)
2022-01-26<EMAIL_ADDRESS>Make sure 'FlutterSplashView.onRestoreInstanceState' doesn't crash (flutter/engine#31045)
2022-01-26<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from zf21TVngZ... to SPvjJJkj6... (flutter/engine#31082)
2022-01-26<EMAIL_ADDRESS>Roll Dart SDK from 8bc8a03ea5cd to 10875d49378b (1 revision) (flutter/engine#31080)
2022-01-26<EMAIL_ADDRESS>Roll Skia from b25fcd8996f1 to 2b80cb6636ef (1 revision) (flutter/engine#31076)
2022-01-26<EMAIL_ADDRESS>Roll Dart SDK from f1083ebd4b92 to 8bc8a03ea5cd (1 revision) (flutter/engine#31077)
2022-01-26<EMAIL_ADDRESS>Roll Skia from b17612940115 to b25fcd8996f1 (1 revision) (flutter/engine#31075)
2022-01-26<EMAIL_ADDRESS>Roll Dart SDK from 9cb9c6d71976 to f1083ebd4b92 (1 revision) (flutter/engine#31073)
2022-01-26<EMAIL_ADDRESS>Roll Skia from 4d1a935835ea to b17612940115 (1 revision) (flutter/engine#31071)
2022-01-26<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from rgD2e4TdC... to zf21TVngZ... (flutter/engine#31070)
2022-01-26<EMAIL_ADDRESS>Add option to enable serial GC (flutter/engine#30975)
2022-01-26<EMAIL_ADDRESS>Remove version flag from pm_tool.gni (flutter/engine#31069)
2022-01-26<EMAIL_ADDRESS>Roll Skia from 9eb1679b77af to 4d1a935835ea (5 revisions) (flutter/engine#31068)
2022-01-26<EMAIL_ADDRESS>[Win32, Keyboard] Store keyboard message session (flutter/engine#31047)
2022-01-26<EMAIL_ADDRESS>Add benchmark for DrawDRRect (flutter/engine#31037)
2022-01-25<EMAIL_ADDRESS>Roll Dart SDK from 42976b233886 to 9cb9c6d71976 (1 revision) (flutter/engine#31067)
2022-01-25<EMAIL_ADDRESS>Roll Skia from f7daa1f50d0b to 9eb1679b77af (3 revisions) (flutter/engine#31066)
2022-01-25<EMAIL_ADDRESS>Roll Skia from 1a9e7531e04a to f7daa1f50d0b (2 revisions) (flutter/engine#31064)
2022-01-25<EMAIL_ADDRESS>Revert "[fuchsia] Switch from core-jit to core snapshots. (#30744)" (flutter/engine#31065)
2022-01-25<EMAIL_ADDRESS>Roll Skia from cc473cd92fc0 to 1a9e7531e04a (6 revisions) (flutter/engine#31060)
2022-01-25<EMAIL_ADDRESS>Roll Dart SDK from fd27fda14d2b to 42976b233886 (1 revision) (flutter/engine#31058)
2022-01-25<EMAIL_ADDRESS>Roll Skia from c5b381c4798c to cc473cd92fc0 (4 revisions) (flutter/engine#31056)
2022-01-25<EMAIL_ADDRESS>Make a python script executable (flutter/engine#31042)
2022-01-25<EMAIL_ADDRESS>[web] Fix screenshot tests running locally (flutter/engine#31054)
2022-01-25<EMAIL_ADDRESS>Roll Skia from 83c85affad78 to c5b381c4798c (2 revisions) (flutter/engine#31053)
2022-01-25<EMAIL_ADDRESS>Roll Skia from 3968ad6301e7 to 83c85affad78 (1 revision) (flutter/engine#31051)
2022-01-25<EMAIL_ADDRESS>Roll Dart SDK from ef3156b1329f to fd27fda14d2b (1 revision) (flutter/engine#31050)
2022-01-25<EMAIL_ADDRESS>Roll Dart SDK from e388130c4109 to ef3156b1329f (1 revision) (flutter/engine#31048)
2022-01-25<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from wr5jUf8jv... to rgD2e4TdC... (flutter/engine#31040)
2022-01-25<EMAIL_ADDRESS>Roll Skia from 42eba20260a8 to 3968ad6301e7 (3 revisions) (flutter/engine#31044)
2022-01-25<EMAIL_ADDRESS>Roll Dart SDK from 4f40923fc27d to e388130c4109 (1 revision) (flutter/engine#31041)
Also rolling transitive DEPS:
fuchsia/sdk/core/mac-amd64 from wr5jUf8jvX6P to SPvjJJkj6m2J
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
...

Commit queue failed; closing this roll.