Tipsy  not HTML5 valid

Hello,
Using Typsy link in webpage's head generate 2 errors :

Attribute name not allowed on element link at this point.
Element link is missing required attribute href.

If I use attribute title instead name, Tipsy plugin don't detect link.
Also, if I use on existing link rel="author" (with both attributes title and name or not) Tipsy plugin don't detect it…

The link element is indeed not the correct element to use for thus purpose. Correct markup should be:
<meta content="Donald Duck"
<EMAIL_ADDRESS>      name="author">

I would advocate publishers go for the <meta /> approach and we should only document that going forward.

Unless you’re aware of massive adoption (…) then just drop support for the link element. No need to slow down every single page view for all eternity to support this given the current level of adoption*.
* assuming it’s low, which I suspect.

My vote is to keep backwards compatibility, even if that only means a handful of sites. I'm fine going with just <meta /> if @karger is in agreement.

Pull request for site documentation + generator as well as the extension coming up (I’ve been poking around in this area of the code tonight, so shouldn’t take more than a few minutes).

I agree that we should move to meta and should change the documentation reflect that.  But I also think having the 2 lines of code needed to support  doesn't hurt anything.  We can remove it from the documentation so no new ones are created, and can remove it from the code in a few years once tipsy is the main payment standard---at that point the few people relying on it will notice and fix if their site stops working.

Here is the implementation I’ll be going with: Authors will be selected first, if that is unavailable try selecting the publisher, and lastly look for the legacy tag.
meta[name=author] || meta[name=publisher] || link[rel=author] || /tipsy.txt

I’ll leave this here to allow for objections, and will submit pull requests for generator+docs and the extension and WP plugin in a day or two.

@da2x I can help rebase anything that gets conflicted btw, so please don't hesitate to open PRs.

@tbranyen (I’m trying to incentivise those of you with commit access to merge my stuff.) Plus, I need some sleep now; and wanted to give people an opportunity to come with suggestions before I did the work.

Add meta support, deprecate the link and remove it in 2 years…

OK, so data-* can’t be used for this purpose.
“These attributes are not intended for use by software that is independent of the site that uses the attributes.” “User agents must not derive any implementation behavior from these attributes or values. Specifications intended for user agents must not define these attributes to have any meaningful values.”
There isn’t another property in HTML that is suitable for custom data, and this can’t be expressed with the current mechanism. Luckily, the HTML+RDFa extension is quite suitable. However, that would create messy parsing and a require increased document complexityand thus add development requirement for publishers.
As a shortcut, which is also the one preferred by Google, is using JSON-LD. Publishers copy and paste a blob anywhere into their documents.
Existing ecommerce and payment vocabularies don’t express a payment method information that is not tied to a transaction. There is also currently no way to express a volunteer supporter/patreon payment scheme. A vocabulary extension would be needed: SupporterPayment, paymentOptions, paymentService, and paymentTarget (defined in http://tipsy.csail.mit.edu/vocab#) .
Websites would insert this blob (in expanded format) instead:
<script type="application/ld+json">
  {
    "@context": "http://schema.org",
    "@type": "WebSite",
    "potentialAction": {
      "@context": {
        "@vocab": "http://tipsy.csail.mit.edu/vocab#"
      },
      "@type":  "SupporterPayment",
      "paymentOptions": [
          {
            "paymentService": "paypal",
            "paymentTarget"<EMAIL_ADDRESS>          },
          {
            "paymentService": "bitcoin",
            "paymentTarget": "3bafbaf"
  } }  ] }
</script>

or alternatively in compacted format:
<script type="application/ld+json">
  {
    "@type": "http://schema.org/WebSite",
    "http://schema.org/potentialAction":
    {
      "@type": "http://tipsy.csail.mit.edu/vocab#SupporterPayment",
      "http://tipsy.csail.mit.edu/vocab#paymentOptions":
        {
          "http://tipsy.csail.mit.edu/vocab#paymentService": "paypal",
          "http://tipsy.csail.mit.edu/vocab#paymentTarget"<EMAIL_ADDRESS>  } } }
</script>

So, not as sexy as a single HTML tag, but there is currently no mechanism in HTML that allows extending it in a “sexy” fashion. RDFa is the preferred way to express invisible metadata, it’s extensible to fit Tipsy’s needs, and it can be expressed as a copy-and-paste drop-in blob for publishers.
This doesn’t express the name of the author or publisher. This can either be derived from the meta tags, or by reading the publisher or author data out of JSON-LD (which is very much recommended by Google these days anyway). HOWEVER, seeming as there is no method for expressing the author/publisher in the /tipsy.txt format, which is strongly preferred by publishers, I’d say this be dropped from the in-document mechanism as well. Unless there is a very strong use-case for expressing per-author payments rather than per-publisher/website (or per-page/URL subfolder as expressed in /tipsy.txt). No current publisher uses anything but "_" (whole-domain) structure, by the by.
Rewriting the generator is like 15 minutes work. The WordPress extension could be done in another 15 minutes, but the extension would probably take me 50 minutes. 10 extra minutes to also support compacted format, although I believe if we only document and generate expanded JSON-LD, than this ought to be enough. I’m just a tad worried that some "page optimizer" somewhere with try to compact the JSON blob to be clever. Add another hour for documentation.
Alternatively, Tipsy could just drop a JSON blob in the window object and be done with it. This requires JavaScript execution on every page load for Tipsy users and non-tipsy users alike — something JSON-LD doesn’t. JSON-LD is thus the better option for the web ecosystem as a whole as it doesn’t degrade website execution performance nor litter the global window object.
<script>
  window.tipsyPaymentOptions = window.tipsyPaymentOptions ||
    [
      { "paypal"<EMAIL_ADDRESS>},
      {"bitcoin": "3bafbaf" } ];
</script>

Thoughts?

What about a custom element?
<tipsy-payment style="display: none"<EMAIL_ADDRESS>/>

[I’ve updated the above JSON-LD proposal to re-use more of existing schemas.]

What about a custom element?

Not a W3C recommendation yet, just in draft state. If I remember correctly, then CSS isn’t applied to unknown [a.ka. custom] elements in older browsers. Introducing such an element could mess up people’s layouts by introducing a block element in unexpected places.
I’m only slightly more in favor of JSON-LD because it potentially makes the payment information available to more user-agents than just Tipsy. Others are less likely to implement and recommend something with a prominent vendor prefix.
I have no strong objections for going with either solutions, though I have a [potentially irrational] feeling that the approach intended for including invisible semantic metadata soups may be better for the web.

So I generally favor adherence to standards but I'm even more strongly in favor of ease of use for the target population and, equally important, not scaring them away with big messy syntax.  Even if it's automatically generated, it will be a problem if they have to see it.  So, for example, I think it would be fine for the wordpress plugin, which embeds the payment info without the author seeing it, to generate json.  But I don't think it's a good idea to ask people manually editing their pages to add script tags and mysterious json.
Presumably, most of the novices will just go with the tipsy.txt file, so they won't be impacted.  But I'd like something tolerable for the remaining novices.  If we have to break standards to do that, I'd advocate that we do.  If we are going to break standards, then doing so in a way that still parses/validates seems preferable.  So, e.g., using the data-* attribute in a meta tag may violate the semantics of data tags, but it is perfectly valid syntax.  So I'd lean towards something like that.
I don't know enough about the relative adoption/understanding of link vs meta tags to have a strong sense of which will seem more natural/appealing to novices---but that's what I'd base the choice on.

I’ve done some testing, and custom elements will cause all sorts of problems with various libraries and HTML processors. No go until support improves. JSON-LD is the correct option, but data-* is the best option.
How about this?
<meta content="Daily Drivel"
      data-payablewith-paypal="merchantidoremail"
      name="publisher">
<meta content="Donald Duck"
      data-payablewith-bitcoin="3bafbaf"
      name="author">

@da2x I like the <meta/> tag better than the alternatives. :+1:

Like the meta too.
Why not add a trick to hide email address in data-* (like replacing @ by something else), for limit abuses by bad crawlers ? ;)

Why not add a trick to hide email address in data-* (like replacing @ by something else), for limit abuses by bad crawlers ? ;)

You’re not supposed to use email addresses anyway. Merchant IDs (required to receive payments via buttons) take the place of email addresses.

But you use merchantidORemail, so Murphy use email. :p

This probably won't matter much for a majority of users. Wikipedia splatters their payment email address everywhere on their site, which is incredibly easy to crawl. For the most part I see email addresses listed as ones that are for payments only. Otherwise if you start getting spam, remove it in favor of a merchant id per what @da2x says. I don't think we should enforce either. /2cents

Bon.
tltr : apply the Murphy's Law > force merchantID or hide email
Beaucoup d'utilisateurs d'internet n'ont pas grande connaissance de ces subtilités, utilisent une seule adresse email pour tout, y compris Paypal pour payer ou recevoir de l'argent.
Si vous voulez que Tipsy soit un outils qui se démocratise rapidement, soit proposé dans les principaux CMS et donc potentiellement utilisé par des bloggeurs rompus à ces pratiques ou complètement amateurs, vous devez considérer que Mr Jean (un noob) utilise son adresse email principale si on lui propose un champ correspondant, et ne se rendra probablement jamais compte que les spams qu'il recevra ensuite soient dû à sa mauvaise utilisation.
C'est pour ça que je pense qu'il faudrait soit forcer l'utilisation du merchantID, soit empêcher que l'adresse email puisse être récupéré.
Sorry for my lack of vocabulary. ><


feat(config): add failOnPendingTests option.

Recently jasmine added support for reporting pending (incl due to fit()) tests.
These appear in results as skipped tests and this failOnPendingTests option allows us to fail tests
with skipped tests.

:white_check_mark: Build karma 2414 completed (commit https://github.com/karma-runner/karma/commit/89a68f110e by @johnjbarton)

:white_check_mark: Build karma 2417 completed (commit https://github.com/karma-runner/karma/commit/6180b2e4a4 by @johnjbarton)


fixed description issue

@llaske this descrption box issue is still present.
Earlier scene :

(committed to dev branch)
Now fixed !!

@llaske this issue is still there.

@llaske whats your take on this.

@llaske this description box issue is still there. Kindly review this.

Hey @DhruvMishra1826,
I suggest going to the path activities/<Any-Activity>/lib/sugar-web/graphics/css/sugar-96dpi.css, choosing the textarea element from the CSS, and appending resize: none; to fix the resizing issue, do this for all the activities of Sugarizer.
I already solved a similar issue in my pull request #1296. You can review it for ideas on how to solve this issue in all Sugarizer activities.
Hope this helps!

Hey @DhruvMishra1826,
I suggest going to the path activities/<Any-Activity>/lib/sugar-web/graphics/css/sugar-96dpi.css, choosing the textarea element from the CSS, and appending resize: none; to fix the resizing issue, do this for all the activities of Sugarizer.
I already solved a similar issue in my pull request #1296. You can review it for ideas on how to solve this issue in all Sugarizer activities.
Hope this helps!

Yes thanks, I know how to do this, but waiting for @llaske views on this.

I think resize: vertical is better since in the Activity palette it could make sense to increase the size vertically.
Plus, even if I'm agree to do the fix in sugar-96dpi.css it will be better to limit the update to the activity palette style to avoid side effect in activity who contains a textarea.

I think resize: vertical is better since in the Activity palette it could make sense to increase the size vertically. Plus, even if I'm agree to do the fix in sugar-96dpi.css it will be better to limit the update to the activity palette style to avoid side effect in activity who contains a textarea.

Related to https://github.com/llaske/sugarizer/issues/1303

Fixed in https://github.com/llaske/sugarizer/pull/1522


Provide immutable default sets

This PR adds new aliases for the default set as it was in the last 4 public releases of the service.  See the change to docs/features.html for more detailed explanation of usage.  This is intended to address recent concerns over the release of v3.6 introducing a bug that broke IE11 even for users that were not requesting the Map and Set polyfills.

One comment about the documentation.
The solution itself is perfect. 👍


[Netty 5] Introduce EventExecutorHandler

Now that we have removed the ability to specify an EventExecutor when adding handlers to the ChannelPipeline we need to provide some replacement functionality.
This is a followup of https://github.com/netty/netty/pull/8778 and related to https://github.com/netty/netty/issues/8512

will start to work on this now

After thinking more and more about this I think the best what we can do is just not add something like this. I detailed the motivation for this in our wiki. Just to make it easier here is it:
### ChannelPipeline.add*(EventExecutorGroup...) removed

In netty 4.x we added the ability to add `ChannelHandler`s to a `ChannelPipeline` with an explicit `EventExecutorGroup`. While this seemed like a good idea it turned out that it has quite some problems when it comes to life-cycles:

- `handlerRemoved(...)` , `handlerAdded(...)` may be called at the "wrong time". This can result in a lot of problems. At worse it could mean that `handlerRemoved(...)` is called and the handler frees some native memory as it expect that the handler will never used anymore. What could happen here is that after it is called `channelRead(...)` may be called which then try to access the previous freed memory and so crash the JVM.
- correctly implement "visibility" in terms of concurrent access / modifications of the pipeline is quite problematic as well.

With this in mind we realised that really what the user mostly want is to have the incoming messages handled by another thread to process the business-logic. This is better be done in a custom implementation provided by the user as the user has a better handle on when things can be destroyed or not.


add test to check data integrity after validation

Q
A

Is bugfix?
no

New feature?
no

Breaks BC?
no

Tests pass?
no

Test for commit https://github.com/yiisoft/yii2/commit/205f16811a6071ad5d1dbea4cba9fd8a3282f6c5 showing the lost of data integrity.

Resolved by commit 2d5a0bf5aeeb3843a9dd3c9c12a6c680035cede5
Thank you.


fix(scripts): make post-checkout executable

file permissions were wrong :(

That warning was so hard to spot, sorry for not picking it up during review :/


Branch 2.1

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.

@lnmohankumar please close this

Can one of the admins verify this patch?

Please close this issue


docs(figma notes): add figma and design notes

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes: #77
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Adds links to Figma and release notes
Type of change:
Please delete options that are not relevant.

[ ] New capability
[ ] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@leeejune this PR is approved to merge. Please sign the CLA.
Will you update the docsite for the new page?

:tada: This PR is included in version 3.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Update macOS browser waitlist

Task/Issue URL: https://app.asana.com/0/414235014887631/1203067395084689/f
Tech Design URL:
CC:
Description:
This PR updates the macOS browser interface.
Note that the notification logic has been left as-is, to allow for users who join the waitlist and then update their app version. We will still be advancing the waitlist for some time, so those users should continue to receive alerts.

Steps to test this PR:

Open settings, and tap on DuckDuckGo Desktop App
Validate that the next screen matches the Figma file found in the Asana task. Note: there is only one screen now, all of the previous waitlist states have been removed.

Screenshots:

:wave: Click here for screenshots

Copy Testing:

[x] Use of correct apostrophes in new copy, ie ’ rather than '

Orientation Testing:

[x] Portrait
[x] Landscape

Device Testing:

[x] iPhone SE (1st Gen)
[x] iPhone 8
[x] iPhone X
[x] iPhone 14 Pro
[x] iPad

OS Testing:

[x] iOS 14
[x] iOS 15
[x] iOS 16

Theme Testing:

[x] Light theme
[x] Dark theme

Internal references:
Software Engineering Expectations
Technical Design Template
When ready for review, remember to post the PR in MM

As a side note - when I was testing on iPad this webpage tries to download the dmg (in both Safari and DuckDuckGo iOS apps) which I'm guessing is not expected behaviour


Lightline Colorscheme

As the title states, would love a lightline colorscheme bundled with the plugin. Thanks in advance.

I dont'use lightline, I build galaxylinewritten in lua and support highly custom better than lightline or
spaceline    written in vimscript.


Feature/Firebase Guardians

In this PR users can:

Invite other users to be their guardians
View and manage a list of guardians.

Multiple new screens and firebase code added. Please run this PR.
Important:  This PR us behind a feature flag. In oder to see the "Guardians Button in your profile you must set the feature flag to true in the default and in the firebase config." Or just hardcode the flag check.

Well done!

Well done!


Moved model_validation tests to invalid_models_tests.

Later I'd rename invalid_models_tests to something like model_checks so it's not suggested that only invalid models can go there, unless anyone has a different idea.

LGTM


Way to run tasks

Hi, I don't know how to run tasks by using this Dockerfile.

Where should I place tasks.py file to register tasks to run?
How can I notice new tasks for this container?

If you know, please tell me.  :smiley:

You can mount your tasks.py into the container in the users home directory (/home/user) and I think use the CELERY_APP environment variable (this worked for me, but I was browsing the issues here hoping to find a list of available environment variables...)
docker run --volume /path/to/tasks.py:/home/user/tasks.py --env CELERY_APP=tasks celery

This needs to be documented.
How can this be used with django for example?

+1 for some documentation :innocent: (what we've got is impossibly sparse right now: https://github.com/docker-library/docs/blob/master/celery/content.md)

that's tricky. i tried with docker-compose but i can't find a way. has anyone been able to run the celery in docker and use the integration with django??

Using this image seems ridiculous. If you have an application container, as you usually have with Django, you need all dependencies (things you import in tasks.py) installed in this container again.
That's why other projects (e.g. cookiecutter-django) reuse the application container for Celery, and only run a different command (command: celery ... worker) against it with docker-compose.

Indeed, Sentry includes a way to run Celery directly as well.
Pondering whether this image actually makes sense to simply deprecate,
similar to the deprecation of the django image (and for similar reasons).

I've opened https://github.com/docker-library/celery/issues/12 to discuss this image's deprecation.  In the meantime, I'm going to close this issue since I think the original question is as solved as it's going to get. :+1:

Agreed, it's not going to be much more difficult to replace this image with a build on a standard Python image with celery added to pip's requirements.txt for example.
Actually, doing so in the first place would have saved me two hours yesterday: This celery docker image ignores the broker url when provided from inside python like so app = Celery('tasks', broker='my url'), and only allows it to be set from the docker env variable CELERY_BROKER_URL. Ok this is kinda hinted in the doc, my bad for not reading carefully. But it's clearly counter-intuitive and time consuming to find this out when having your nose in the code and not expecting some of the parameters to be forced but not the others (the backend argument is not forced like this for example, although it has the same kind of properties that could refer to other docker containers: host, port, ...).
Anyway I'm rambling again, I came here to comment about this parameters forcing but since the whole thing is deprecated I reckon there's no need.


where 'status' = ?

Summary of problem or feature request
This happen when data not found
using where (int) become ? (question mark) not showing real number,
is this feature or to hide real number ?
Code snippet of problem
Only status
$members = Members::where('status',1);
		return Datatables::of($members)
			->addColumn('action', 'backend.pages.members.buttons')
			->rawColumns(['action'])
			->addIndexColumn()
			->make(true);

result
{"draw":0,"recordsTotal":0,"recordsFiltered":0,"data":[],"queries":[{"query":"select count(*) as aggregate from (select '1' as `row_count` from `members` where `status` = ?) count_row_table","bindings":[1],"time":0.4},{"query":"select * from `members` where `status` = ?","bindings":[1],"time":0.3}],"input":[]}

add orderby hope it well
$members = Members::where('status',1)->orderBy('name','ASC');
		return Datatables::of($members)
			->addColumn('action', 'backend.pages.members.buttons')
			->rawColumns(['action'])
			->addIndexColumn()
			->make(true);

result
{"draw":0,"recordsTotal":0,"recordsFiltered":0,"data":[],"queries":[{"query":"select count(*) as aggregate from (select * from `members` where `status` = ? order by `name` asc) count_row_table","bindings":[1],"time":0.37},{"query":"select * from `members` where `status` = ? order by `name` asc","bindings":[1],"time":0.37}],"input":[]}

using DB
$members = DB::table('members')
						->select('*')
						->where('status',1)
						->get();

result
{"draw":0,"recordsTotal":0,"recordsFiltered":0,"data":[],"input":[]}

-->
System details

Operating System Ubuntu 16.04
PHP Version 7.2.10
Laravel Version 5.6
Laravel-Datatables Version ~8.0

i forgot to write my question,  can result from eloquent become empty like this, if null / empty :
{"draw":0,"recordsTotal":0,"recordsFiltered":0,"data":[],"input":[]}

This is not an issue and should not be affecting your results. The ? is just a placeholder, but if you look in the queries data, you will also see bindings which is what is placed in the ? when it is run as SQL.
For example {"query":"select * from memberswherestatus= ? order byname asc","bindings":[1]} becomes SELECT * FROM members where status = 1 order by name asc

@oranges13 is correct, it's just a placeholder for query bindings which is basically a protection from an SQL Injection attacks.


Missing menu item Using sass at document

Missing "Using Sass" at main document

Not sure what you mean exactly, can you please elaborate on what the issue is?

Hi @samit4me. At main document

go to folder docs
go to "Table of Contents" > section CSS. You will see 2 items: syled-components, sanitize.css
Go to section CSS. Now it turns into 3 items: syled-components, sanitize.css, Using Sass.

I see, it's missing from the Table of contents, thanks for reporting the issue, nice find 👌. Are you keen to open a PR? If so, please take a look at the CONTRIBUTING guidelines. If not, that's cool too, either way, please let us know so it can get fixed quickly.

I already looked at it. And I have made a PR followed this guildline

Here is PR:

Thank you for that, linking the issue in the PR is useful to prevent this confusion.

Hi @samit4me. Here is another PR I have made to branch dev
https://github.com/mxstbr/react-boilerplate/pull/1271


Support for Polymer 2.0

Any plans to support Polymer 2.0, either via hybrid-mode or directly?

We'll discuss it when 2.0 becomes official release (it is still RC). I heard about Polymer team plans to offer built-in solution for data architecture in 2.0 at some point. We will see how it goes.

@militeev Now when 2.0 is released, do you have any update on this?

@militeev Polymer team plans to offer built-in solution for data architecture in 2.0? Sounds really interesting!


Problems: embed the search box into the header

Instead of showing an action to enable the "type to filter", why do we not just embed an input field in the header that you can just start typing?
It is possible to contribute actions to the header that heave their own UI. E.g. the dropdown in the debug viewlet or output channel is done like that:

@bpasero That's a good idea.. this would also fix some usability issues. Thanks.


chore: update saved search tests

The type of this PR is: Chore
Update saved search tests

#squashongreen


Scheduler for off-chain workers execution

Offchain workshop follow up task.
There is a common pattern emerging in the way we run offchain workers. Usually we want them to be triggered:

After a block number set in the storage
Only once after that threshold,
But retry after the previous run does not seem to complete.
Randomly for different validators, etc

That could be easily made into a re-usable piece of code that would schedule the runs. Rough (and incomplete) proposal from @kianenigma
fn offchain_worker() {
	// simplest case. It basically adds a `should_run` impl that always return true
	let OffchainLimit::default()::build();

	// Run every N block
	let OffchainLimit::default()::with_interval(10)

	// We can also implement maybe your idea of
	let OffchainLimit::default().with_interval(10).randomise(2)

        /// after some point in time
	let OffchainLimit::after(Storage::get(()).with_interval(10).randomise(2)

	// Any more attributes. I really think the idea of _composability_ can be very neat here, which
	// the builder pattern does very well.

Is this to be grabbed or is @bkchr working on it?

No, you can work on this.

Adding this to the NPoS project as it will (with a very high probability) be used by the OCW workers there.


no need to unpack since license is bundled in recipe [skip ci] [ci skip]

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-webservice.
I tried to re-render for you, but it looks like there was nothing to do.


make compatible with bookworm

The build works with bookworm, but the issue is, that it does not boot complaining about no partition found with PARTUUID from cmdline.txt.
I was debugging the issue down to:

PARTUUID in cmdline is correct after image written
PARTUUID seems to be change by resizing the partition.
This seems to be done pre booting the root partition from within initramfs see https://github.com/RPi-Distro/raspberrypi-sys-mods/tree/bookworm/initramfs-tools/scripts/local-premount
This may be the issue. But this file is not found on the local filesystem. Perhaps recreation of initramfs with adaptions her are requiered.

still not working 100%. Blackscreen after first boot, than works after reboot.
Perhaps this is only an issue with raspi3 or the pi-top, I use for testing


Remove templates

Resolves #160
Readme: Fixed broken badges and links pointing to previous 4.x version

Includes changes from https://github.com/futuredapp/arkitekt/pull/165


Sublime Text, Command-Shift-Period tag closing

(Feel free to just close this, as it's not really a bug, just a wish.)
When writing HTML in Sublime Text Command-Shift-Period just closes the closest tag that it can find that's still open. It's probably my most-used command.
The close-tag demo is pretty cool, but it's more heavy-handed as it's automated insertion rather than on-demand insertion.

Hi Chris, with the attached patch, you can bind keys to "closeTag" to get the behavior you describe. It was a small change to have the addon also expose a command.

Works wonderfully, thank you so much!


Function type definition with anonymous arguments and optional parentheses for single argument

Suggestion

🔍 Search Terms
anonymous type argument, optional parentheses

List of keywords you searched for before creating this issue. Write them down here so that others can find this suggestion more easily and help provide feedback.
✅ Viability Checklist

My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion
Like a anonymous function and anonymous type, it could be nice a type definition with anonymous arguments. Also, as in function definition, function type definition with single argument, the parentheses could be optional.
📃 Motivating Example
This simple enhancement let a more clean and lean code syntaxe.
💻 Use Cases
// currently
function f(callback:(...args: string[]) => void) { ... }

// anonymous arguments
function f(callback: (...string[]) => void) { ... }

// anonymous argument + optional parentheses
function f(callback: ...string[] => void) { ... }

I’m almost positive this is a duplicate.

Me too, but this kind of thing is impossible to search for :(


Builds on windows-latest failing

Recently returned to a project after about a month or so and my windows build is failing all of the sudden with no significant changes made. We can build successfully on hardware.
I'm receiving the following error:

And here is my action:
name: "publish"
on:
  push:
    branches:
      - release

jobs:
  publish-tauri:
    strategy:
      fail-fast: false
      matrix:
        settings:
          - platform: "macos-latest"
            args: "--target aarch64-apple-darwin"
          - platform: "macos-latest"
            args: "--target x86_64-apple-darwin"
          - platform: "windows-latest"

    runs-on: ${{ matrix.settings.platform }}
    environment: Caspr-Releases
    steps:
      - uses: actions/checkout@v3
      - name: setup node
        uses: actions/setup-node@v3
        with:
          node-version: 18.15.0
      - name: install Rust toolchain
        uses: dtolnay/rust-toolchain@stable
        with:
          targets: aarch64-apple-darwin
      - name: install app dependencies and build it
        run: npm install && npm run build
      - uses: tauri-apps/tauri-action@v0
        env:
          various secrets
        with:
          tagName: app-v__VERSION__ # the action automatically replaces \_\_VERSION\_\_ with the app version
          releaseName: "Caspr v__VERSION__"
          releaseBody: "See the assets to download this version and install."
          releaseDraft: true
          prerelease: false
          includeUpdaterJson: true
          args: ${{matrix.settings.args}}

I've found this Stack Overflow answer  that explains that it could have something to do with the tokio runtime version so if I could get some help setting that in our Tauri App that would be great!
Thanks

Can you try a simple cargo update? It should update the version of tokio used in the lockfile.

@FabianLars ran an update and got the same result unfortunately. Just to provide some more context, our MacOS jobs build successfully.

alrighty, i don't see why the SO answer would behave any different (doesn't make much sense to me but maybe cargo is a bit weird here or i just don't get it), but what they are talking about is adding this to the [dependencies] section in your Cargo.toml file
tokio = "1.28"

No luck, it's pretty hard to tell what the actual issue is here, I've tried changing my rust toolchain version, reverting to windows-2019 runtime. My guess maybe is that it would probably have something to do with the toolchain's stable version changing over time?? I'm pretty new to actions and rust tbh. If you think it might be something different I'm open to ideas.
I appreciate your time @FabianLars keep up the good work!

Can you share the list of your rust dependencies so i can try to reproduce it?

My guess maybe is that it would probably have something to do with the toolchain's stable version changing over time?

Yeah, the SO answer should be accurate about that. Not the first time i see this error i think.

Sure thing, I included the added Tokio dependency just FYI
[build-dependencies]
tauri-build = { version = "1.2.1", features = [] }

[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
tauri = { version = "1.2.4", features = ["cli", "dialog-ask", "dialog-confirm", "global-shortcut-all", "macos-private-api", "process-exit", "system-tray", "updater", "window-all"] }
url = "2.3.1"
tauri-plugin-autostart = { git = "https://github.com/tauri-apps/plugins-workspace", branch = "dev" }
tauri-plugin-highlander = { git = "https://github.com/TMRRWinc/tauri-plugin-highlander", branch = "main" }
enigo = "0.0.14"
window-shadows = "0.2.1"
tauri-plugin-store = { git = "https://github.com/tauri-apps/plugins-workspace", branch = "v1" }
sentry = "0.27"
sentry-tauri = "0.1"
dark-light = "1.0.0"
tokio = "1.28"

[dependencies.tauri-plugin-sql]
git = "https://github.com/tauri-apps/plugins-workspace"
branch = "dev"
features = ["sqlite"]

[dependencies.fix-path-env]
git = "https://github.com/tauri-apps/fix-path-env-rs"
branch = "dev"

is it possible for you to swich tauri-plugin-highlander for tauri-plugin-single-instance? https://github.com/tauri-apps/plugins-workspace/tree/v1/plugins/single-instance
highlander uses an outdated version of sysinfo which in turn uses the problematic version of ntapi

Highlander is in fact the culprit thanks again! @FabianLars


Google Analytics improvements

Already have this roughly working. However one issue seems to be that it is tracking page titles rather than page urls at least on evidence of web3 site (? to be confirmed)
Acceptance

[x] Setup

[x] Tracking page urls (instead of or as well as page titles) ✅2022-07-22 INVALID - actually tracking page urls. Just that page names show by default. See screenshost just below showing the ability to select page urls in GA

[x] Documented
[x] Deployed on flowershop.app ✅2022-07-21 and tested it worked 2022-07-22

Tasks

[x] Get code
[x] Add to config
[x] Deploy
[x] Test

FIXED. Deployed and working and nothing to fix in the setup.


Create an object from another object

Lets say we have a user -> groups mapping table.  It would be useful to embed some or all of the group information in the mapping table.  This allows us to avoid doing additional queries for group information in many circumstances.
class Group(Model):
    group_id = cqlengine.UUID
    name = cqlengine.Text() # part of the
    num_members = cqlengine.Integer()

class UserGroupIndex(Model):
    user_id = cqlengine.UUID(primary_key=True)
    group_id = cqlengine.UUID()
    name = cqlengine.Text() # part of the

There's a lot of use cases for creating Group objects which are partially hydrated from another model.
Some potential ideas for how this could look from a user perspective:
def get_teams(user_id):
    return [x.team for x in UserTeamIndex.objects(user_id=user_id)]

# vs

def get_teams(team_id):
    return UserTeamIndex.objects(user_id=user_id).masquerade_as(Team)

It's important to keep in mind that the objects (in this case, Group) might only be a partial object, and not have all the fields.  We need to figure out how to resolve this problem.

We are migrating all Github issues to the public Jira. Please find this issue in Jira:https://datastax-oss.atlassian.net/browse/PYTHON-271


Fix spacing of stacked custom radios & checkboxes

Fixes #17432.

Going to hold off on this for now—might try a different method here all together.

Superseded by #20671 alongside some other changes to how those elements are built. Thanks though!


Ubuntu installation error

I keep getting the following error:
        activesupport requires Ruby version >= 2.2.2```

My ruby version is:
`ruby 2.2.4p230 (2015-12-16 revision 53155) [x86_64-linux-gnu]`

What seems to be the problem then?

Fixed it by separately installing activesupport

gem install activesupport


Add CodeQL

Fix #107

Merging. We can always remove this action if we have any frustrations with it :)


CSR allocation (and tentative address): mclicbase

Current documentation states that mclicbase CSR would have an address of 0x3?? but 0x3?? indicates machine-mode read-write CSRs.  It seems inappropriate for a read-only CSR.
Also, similar general purpose configuration data structure-pointing CSR mconfigptr (a part of the privileged architecture version 1.12) has an address of 0xf15.
If this CSR is going to be preserved, 0xf?? would be better than the current tentative address 0x3??.
Strictly speaking, xintstatus are also read-only but their addresses are okay to me (represents dynamic state information and updated by other CSR updates).

We are planning on moving these to parameters instead of CSRs.  related to closed issue #80.
The mclicbase register and clicinfo are likely to be replaced by the general discovery mechanism that is in development.
But we will leave this issue open until that is done.   your point about addresses is taken.  similar to #88 where intstatus are read-only but currently allocated in the r/w CSR range.

These should be reallocated to appropriate CSR ranges.

Hmm, xintstatus also need to change (as pointed out in #88) considering CSRs containing dynamic state information like vl and vtype from V extension uses 0xC??, a URO CSR address range.
I confirmed a PR which removes mclicbase (which is okay).

Closed with https://github.com/riscv/riscv-fast-interrupt/pull/284 and https://github.com/riscv/riscv-fast-interrupt/pull/285


The push notification package display name Change to MOBILE PUSH

@starpit
I have noticed that the push notification package display name is changed to `MOBILE PUSH`. The name should be `PUSH NOTIFICATIONS`.

in discussion with @sjfink we've decided to use this name temporarily in the UI, until we can find a way to display longer names. "Push Notifications" is quite long.


initial b070

@marie135 J2 has 5v connected but not 3.3, please add an LDO (you can copy off of b002) to supply 3.3v.


Decode SendGrid adapter 4xx response

👋 Thanks for the library, it's very well put together.
In the SendGrid API it provides error messages for fields, or even general messages but this is not decoded.
This pull request adds decoding for 4xx responses, so that when using the Sendgrid adapter, you can easily use the body from the response.

Thanks for the PR. I made a minor change to make the status code boundaries more consistent. I'll merge on CI pass.

@jackmarchant are you waiting on this to be released?

No I’m not waiting for it. Just saw it when looking through


New Tractus-X project contributor

Issue Detail
I'm an active contributor to eclipse already. For some reasons our devsecops team can't able add contributor role to my account.
I have already signed ECA. Both GitHub and Eclipse account have the same email address as primary.
Here is my project/contribution - managed-simple-data-exchanger-frontend
Necessary Information
Email address to add<EMAIL_ADDRESS>GitHub user id: ajithsimons
Committers, that could take over: @FaGru3n @almadigabor

hi @ajithsimons, I've added you to project contributors, pls check and confirm.


Can’t assign values to mixed when it’s within an Array or Object type

Flow version: v0.128.0
Expected behavior
According to the documentation page Mixed Types, anything can be assigned to mixed:

mixed will accept any type of value. Strings, numbers, objects, functions– anything will work.

That documentation page doesn’t say it should work any different when mixed is inside an array or object. Therefore, the following code, which uses an Array<mixed> type, should pass Flow type checking:
const arrayOfNumbers: Array<number> = [1, 2, 3];
const arrayOfMixed: Array<mixed> = arrayOfNumbers;

Additionally, all of the example code in the “Actual behavior” section should pass Flow type checking.
I know that covariance and contravariance can sometimes cause similar confusion, but I don’t see why it would change my expectations here.
Actual behavior
Flow raises type errors in four of the six paragraphs of code below (demo on Try Flow):
// @flow

// mixed works when it’s a top-level value
const num: number = 3;
const mix: mixed = num;

// mixed FAILS inside array types
const arrayOfNumbers: Array<number> = [1, 2, 3];
const arrayOfMixed: Array<mixed> = arrayOfNumbers;

// mixed FAILS inside object types
const personWithNumberAge: { age: number } = { age: 55 };
const personWithMixedAge: { age: mixed } = personWithNumberAge;

// mixed FAILS inside indexed object types
const stringsToNumbers: { [string]: number } = { "a": 1, "b": 2 };
const stringsToMixed: { [string]: mixed } = stringsToNumbers;

// mixed FAILS inside indexed object types as function parameters (my use-case)
function withStringsToMixed(obj: { [string]: mixed }): Array<string> {
  return Object.keys(obj);
}
withStringsToMixed(stringsToNumbers);

// generics work inside indexed object types
function withStringsToGeneric<T>(obj: { [string]: T }): Array<string> {
  return Object.keys(obj);
}
withStringsToGeneric(stringsToNumbers);

The type errors are of this form:

Cannot assign somethingContainingNumbers to somethingContainingMixed because number [1] is incompatible with mixed [2] in [some container]. [incompatible-type]

All four errors that Flow raises for the above example code:
    9: const arrayOfMixed: Array<mixed> = arrayOfNumbers;
                                          ^ Cannot assign `arrayOfNumbers` to `arrayOfMixed` because number [1] is incompatible with mixed [2] in array element. [incompatible-type]
        References:
        8: const arrayOfNumbers: Array<number> = [1, 2, 3];
                                       ^ [1]
        9: const arrayOfMixed: Array<mixed> = arrayOfNumbers;
                                     ^ [2]
    13: const personWithMixedAge: { age: mixed } = personWithNumberAge;
                                                   ^ Cannot assign `personWithNumberAge` to `personWithMixedAge` because number [1] is incompatible with mixed [2] in property `age`. [incompatible-type]
        References:
        12: const personWithNumberAge: { age: number } = { age: 55 };
                                              ^ [1]
        13: const personWithMixedAge: { age: mixed } = personWithNumberAge;
                                             ^ [2]
    17: const stringsToMixed: { [string]: mixed } = stringsToNumbers;
                                                    ^ Cannot assign `stringsToNumbers` to `stringsToMixed` because number [1] is incompatible with mixed [2] in the indexer property. [incompatible-type]
        References:
        16: const stringsToNumbers: { [string]: number } = { "a": 1, "b": 2 };
                                                ^ [1]
        17: const stringsToMixed: { [string]: mixed } = stringsToNumbers;
                                              ^ [2]
    23: withStringsToMixed(stringsToNumbers);
                           ^ Cannot call `withStringsToMixed` with `stringsToNumbers` bound to `obj` because number [1] is incompatible with mixed [2] in the indexer property. [incompatible-call]
        References:
        16: const stringsToNumbers: { [string]: number } = { "a": 1, "b": 2 };
                                                ^ [1]
        20: function withStringsToMixed(obj: { [string]: mixed }): Array<string> {
                                                         ^ [2]

The last paragraph of my code example, “generics work inside indexed object types”, shows that this issue can be worked around by adding an unused generic parameter to the function. This workaround is only possible if the mixed-containing type is a parameter to the function, not if it’s a local variable.
For comparison, when I try writing equivalent code in TypeScript by changing mixed to unknown, TypeScript raises no errors. I know that Flow and TypeScript are not always directly comparable, though, since their type systems differ in a few ways.

I know that covariance and contravariance can sometimes cause similar confusion, but I don’t see why it would change my expectations here.

Here is why:
const arrayOfNumbers: Array<number> = [1, 2, 3];
const arrayOfMixed: Array<mixed> = arrayOfNumbers;
arrayOfMixed[0] = 'foo'; // allowed for `Array<mixed>` but violates `Array<number>`

Because arrayOfMixed is allowed to be mutated and because it is referentially the same array as arrayOfNumbers it is possible that line 3 could occur and is thus an error to allow line 2. There are a couple ways around this.
// use spread to create a new reference which can be mutated without affecting the `arrayOfNumbers`
const arrayOfMixed: Array<mixed> = [...arrayOfNumbers];

// use a read-only array to prevent mutation
const arrayOfMixed: $ReadOnlyArray<mixed> = arrayOfNumbers;

In your follow up examples:
// mixed works when it’s a top-level value
const num: number = 3;
const mix: mixed = num;
// This is allowed because they are constants and can never be reassigned or mutated.

// mixed FAILS inside object types
const personWithNumberAge: { age: number } = { age: 55 };
const personWithMixedAge: { age: mixed } = personWithNumberAge;
// This fails otherwise I could do
// personWithMixedAge.age = 'foo';

Your try flow link, but with all read-only and no errors


implement trie based wildcard support

See #684 #676 #685 #681
We have topic_name and topic_filter validatiton functions. They are already been meged.
I already merged tire supporting functions.
Now, it is a time to implement trie version of wildcard support to test_broker.
@kleunen , could you demonstrate how to use subscription_map and retained_topic map ?
Could you write a code example or pull request ?
I'm not sure sub_con_online and sub_con_offline need to be shared_ptr or not. I remember your previous approach uses shared_ptr for them. Please check it is still required in the current master.
I guess that you replace
https://github.com/redboltz/mqtt_cpp/blob/d035c5bca19eaedd72a05c9582bf23fd6ae3d83a/test/test_broker.hpp#L1699-L1700
to subscription_map.
I think that subscription_map needs to have search by the address of endpoint_t and client_id.
Consider the following scenario.

client1 connected to the broker with CleanStart:false SessionExpiryInterval:inifinity
client1 subscribed topic1. Broker updates subscription_map.
client1 disconnected. Broker need to search subscription_map by the address of endpoint_t and move it to offline one.

It is implemented using multi_index now. See tag_con.
https://github.com/redboltz/mqtt_cpp/blob/d035c5bca19eaedd72a05c9582bf23fd6ae3d83a/test/test_broker.hpp#L1554-L1557
There are some kind of events to be a trigger to access to subscription_map.

publish: access to subscription_map by topic_name and get a set of matched sub_con_online or sub_con_offline.
subscribe: insert to subscription_map.
unsubscribe: remove from subscription_map. The key is topic_filter.
disconnect: move or update sub_con_online element in subscription_map. The key is the address of endpoint_t.
connect: move or update sub_con_offine element in subscription_map. The key is client_id.
timeout: Session expire case. remove sub_con_offline from subscription_map. The key is client_id.

@kleunen , could you tell me how to implement them ?

@kleunen , could you demonstrate how to use subscription_map and retained_topic map ?

Yes, i have an example broker and a small unit test setup which shows how the maps should be used. How can I send you the code example ?

Please put the code here with cpp decoration as follows:
// your code example
int main() {
    subscription_map m;
    ...
}

Tsumaranai monodesuga ( hope this is right thing to say :D )
It does say:
Remaining size should be 1 (root element only)
Remaining size: 4
So something is wrong somewhere, i think this is in the test actually, not the map itself.
//
// Created by wkl04 on 10-12-2019.
//
#include "precomp.h"

#include "subscription_map.h"
#include "retained_topic_map.h"

#include <iostream>
#include <chrono>

void TestSingleSubscription()
{
    std::string text = "example/test/A";

    single_subscription_map< std::function< void() > > map;
    map.insert("example/test/A", [] { std::cout << "example/test/A: Subscriber 1" << std::endl; } );

    try {
        map.insert("example/test/A", [] { std::cout << "example/test/A: Subscriber 2" << std::endl; } );
    } catch(std::exception &e)
    {
        // Should fail due to duplicate subscription
        std::cout << e.what() << std::endl;
    }

    auto handle = map.insert("example/test/B", [] { std::cout << "example/test/B" << std::endl; } );
    std::cout << "Handle of new subscription: " << map.handle_to_subscription(handle) << std::endl;
    map.update(handle, [] { std::cout << "example/test/B updated" << std::endl; } );
   // map.remove(handle);

    map.insert("example/+/A", [] { std::cout << "example/+/A: Plus" << std::endl; } );
    map.insert("example/#", [] { std::cout << "example/#" << std::endl; } );

    std::cout << "Test with hash, plus and full path" << std::endl;
    map.find("example/test/A", [](std::function< void() >  const &a) {
        a();
    });

    map.remove("example/+/A");

    std::cout << "Test with hash, full path" << std::endl;
    map.find("example/test/A", [](std::function< void() > const &a) {
        a();
    });

    map.remove("example/#");

    std::cout << "Test with full path only" << std::endl;
    map.find("example/test/A", [](std::function< void() >  const &a) {
        a();
    });

    map.remove("example/test/A");
    // map.remove("example/test/A");
    map.remove("example/test/B");

    std::cout << "No entries exist anymore" << std::endl;
    map.find("example/test/A", [](std::function< void() >  const &a) {
        a();
    });

    std::cout << "Remaining size should be 1 (root element only)" << std::endl;
    std::cout << "Remaining size: " << map.size() << std::endl;
}

void TestMultipleSubscription()
{
    std::string text = "example/test/A";

    multiple_subscription_map<std::string> map;

    try {
        map.insert("doubleslash//#", "Should fail");
    } catch(std::exception &e)
    {
        // Should fail due to empty path usage
        std::cout << e.what() << std::endl;
    }

    map.insert("example/test/A", "example/test/A: Subscriber 1");
    map.insert("example/test/A", "example/test/A: Subscriber 2");
    map.insert("example/test/B", "example/test/B");
    map.insert("example/+/A", "example/+/A: Plus");
    map.insert("example/#", "example/#");
   // map.insert("#", "Root hash");

    std::cout << "Lookup handle of an entry" << std::endl;
    std::cout << "Size: " << map.lookup("example/test/B").size() << std::endl;
    std::cout << "Path: " << map.handle_to_subscription(map.lookup("example/test/B")) << std::endl;

    map.remove(map.lookup("example/test/A"), "example/test/A: Subscriber 1");

    std::cout << "Test with hash, plus and full path" << std::endl;
    map.find("example/test/A", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    /* map.remove("example/+/A", "example/+/A: Plus");

    std::cout << "Test with hash, full path" << std::endl;
    map.find("example/test/A", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    map.remove("example/#", "example/#");

    std::cout << "Test with full path only" << std::endl;
    map.find("example/test/A", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    map.remove("example/test/A", "example/test/A: Subscriber 1");
    map.remove("example/test/A", "example/test/A: Subscriber 2");
    map.remove("example/test/B", "example/test/B");

    std::cout << "No entries exist anymore" << std::endl;
    map.find("example/test/A", [](std::string const &a) {
        std::cout << a << std::endl;
    }); */

    std::cout << "Remaining size should be 1 (root element only)" << std::endl;
    std::cout << "Remaining size: " << map.size() << std::endl;
}

void TestRetainedTopics()
{
    retained_topic_map<std::string> map;
    map.insert_or_update("example/test/A", "example/test/A: Content 1");
    map.insert_or_update("example/test/B", "example/test/B: Content 1");
    map.insert_or_update("example/A/test", "example/A/test: Content 1");
    map.insert_or_update("example/B/test", "example/B/test: Content 1");

    /*map.find("example/test/A", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    map.find("example/test/B", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    map.find("example/test/+", [](std::string const &a) {
        std::cout << a << std::endl;
    });

    map.find("example/+/B", [](std::string const &a) {
        std::cout << a << std::endl;
    }); */

    std::cout << "Match: example/#" << std::endl;
    map.find("example/#", [](std::string const &a) {
        std::cout << a << " " << a.size() << std::endl;
    });
    std::cout << "Done" << std::endl;

    map.remove("example/test/A");
    map.find("example/+/A", [](std::string const &a) {
        std::cout << a << std::endl;
    });

}

int main(int, char**)
{
    try {
        TestSingleSubscription();
        TestMultipleSubscription();
        TestRetainedTopics();

    } catch(std::exception &e)
    {
        std::cout << e.what() << std::endl;
    }
}
``

And the small broker I made:
#include "precomp.h"

#include <string>
#include <iostream>

#include "mqtt_server_cpp.hpp"
#include "subscription_map.h"
#include "retained_topic_map.h"

using con_t = MQTT_NS::server<>::endpoint_t;
using con_sp_t = std::shared_ptr<con_t>;

struct session_t;
using session_ptr_t = std::shared_ptr<session_t>;
using session_weak_ptr_t = std::weak_ptr<session_t>;

using session_map_t = std::map<MQTT_NS::buffer, session_ptr_t>;

struct session_subscription_t
{
    session_ptr_t session;
    MQTT_NS::qos qos;

    session_subscription_t(const session_ptr_t &session, MQTT_NS::qos qos = MQTT_NS::qos())
        : session(session), qos(qos)
    { }
};

inline bool operator<(session_subscription_t const &a, session_subscription_t const &b)
{
    return a.session < b.session;
}

using subscription_map_t = multiple_subscription_map< session_subscription_t >;

// Map of all subscription -> session
subscription_map_t subs_map;

struct session_t
{
    MQTT_NS::buffer client_id;
    std::weak_ptr<con_t> con;
    std::map<MQTT_NS::buffer, subscription_map_t::handle> subscriptions;

    session_t(const std::weak_ptr<con_t> &con)
        : con(con)
    { }

    ~session_t()
    {
        std::cout << "Release session: " << client_id << std::endl;
    }

    std::shared_ptr<con_t> get_connection()
    {
        auto sp = con.lock();
        BOOST_ASSERT(sp);
        return sp;
    }

    void add_subscription(MQTT_NS::buffer topic, subscription_map_t::handle const &handle)
    {
        subscriptions[topic] = handle;
    }

    subscription_map_t::handle get_subscription(MQTT_NS::buffer topic)
    {
        auto sub_iter = subscriptions.find(topic);
        if(sub_iter == subscriptions.end())
            return subscription_map_t::handle();
        else
            return sub_iter->second;
    }

    subscription_map_t::handle remove_subscription(MQTT_NS::buffer topic)
    {
        auto sub_iter = subscriptions.find(topic);
        if(sub_iter == subscriptions.end())
            return subscription_map_t::handle();

        auto result = sub_iter->second;
        subscriptions.erase(sub_iter);
        return result;
    }

    void publish(MQTT_NS::buffer topic_name, MQTT_NS::buffer contents, MQTT_NS::publish_options options) {
        auto sp = con.lock();
        if(sp) {
            sp->async_publish(
                    boost::asio::buffer(topic_name),
                    boost::asio::buffer(contents),
                    std::make_pair(topic_name, contents), options);
        }
    }
};

// All stored retained topics
retained_topic_map< std::pair<MQTT_NS::buffer, MQTT_NS::buffer> > retained_map;

// Map of all sessions
session_map_t sessions;

inline void close_session(session_ptr_t session) {
    // Remove all subscriptions for this session
    for(auto const &i: session->subscriptions) {
        subs_map.remove(i.second, session_subscription_t(session));
    }

    // Remove the client id from the list of active sessions
    auto i = sessions.find(session->client_id);
    if(i != sessions.end() && i->second == session)
        sessions.erase(i);

    std::cout << "Active sessions: " << sessions.size() << std::endl;
    std::cout << "Subs map size: " << subs_map.size() << std::endl;
}

int main(int argc, char** argv) {
    if (argc != 2) {
        std::cout << argv[0] << " port" << std::endl;
        return -1;
    }
    boost::asio::io_context ioc;

    auto server_endpoint = boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(),boost::lexical_cast<std::uint16_t>(argv[1]));
    auto s = MQTT_NS::server<>(server_endpoint, ioc);

    s.set_error_handler(
            [](MQTT_NS::error_code ec) {
                std::cout << "error: " << ec.message() << std::endl;
            }
    );

    s.set_accept_handler(
            [](con_sp_t spep) {
                auto& ep = *spep;

                session_ptr_t session = std::make_shared<session_t>(std::weak_ptr<con_t>(spep));

                using packet_id_t = typename std::remove_reference_t<decltype(ep)>::packet_id_t;
                std::cout << "accept" << std::endl;

                // Pass spep to keep lifetime.
                // It makes sure wp.lock() never return nullptr in the handlers below
                // including close_handler and error_handler.
                ep.start_session(session);

                // set connection (lower than MQTT) level handlers
                ep.set_close_handler(
                        [session]() {
                            std::cout << "[server] closed session: " << session->client_id << std::endl;
                            close_session(session);
                        });

                ep.set_error_handler(
                        [session](MQTT_NS::error_code ec) {
                            std::cout << "[server] error: " << ec.message() << " " << session->client_id << std::endl;
                            close_session(session);
                        });

                // set MQTT level handlers
                ep.set_connect_handler(
                        [session](MQTT_NS::buffer client_id, MQTT_NS::optional<MQTT_NS::buffer> username, MQTT_NS::optional<MQTT_NS::buffer> password, MQTT_NS::optional<MQTT_NS::will>, bool clean_session, std::uint16_t keep_alive) {
                            auto sp = session->get_connection();

                            using namespace MQTT_NS::literals;
                            std::cout << "[server] client_id    : " << client_id << std::endl;
                            std::cout << "[server] username     : " << (username ? username.value() : "none"_mb) << std::endl;
                            std::cout << "[server] password     : " << (password ? password.value() : "none"_mb) << std::endl;
                            std::cout << "[server] clean_session: " << std::boolalpha << clean_session << std::endl;
                            std::cout << "[server] keep_alive   : " << keep_alive << std::endl;

                            if(sessions.find(client_id) != sessions.end())
                            {
                                // A session with this client id already exists
                                sp->connack(false, MQTT_NS::connect_return_code::identifier_rejected);
                                return false;
                            }

                            session->client_id = client_id;
                            sessions[client_id] = session;

                            sp->connack(false, MQTT_NS::connect_return_code::accepted);
                            return true;
                        }
                );

                ep.set_pingreq_handler([session]() {
                    auto sp = session->get_connection();
                    sp->async_pingresp();
                    return true;
                });

                ep.set_disconnect_handler(
                        [session]() {
                            auto sp = session->get_connection();
                            std::cout << "[server] disconnect received." << std::endl;
                            close_session(session);
                        });

                ep.set_puback_handler(
                        [](packet_id_t packet_id){
                          //  std::cout << "[server] puback received. packet_id: " << packet_id << std::endl;
                            return true;
                        });

                ep.set_pubrec_handler(
                        [](packet_id_t packet_id){
                          //  std::cout << "[server] pubrec received. packet_id: " << packet_id << std::endl;
                            return true;
                        });

                ep.set_pubrel_handler(
                        [](packet_id_t packet_id){
//                            std::cout << "[server] pubrel received. packet_id: " << packet_id << std::endl;
                            return true;
                        });

                ep.set_pubcomp_handler(
                        [](packet_id_t packet_id){
                           // std::cout << "[server] pubcomp received. packet_id: " << packet_id << std::endl;
                            return true;
                        });

                ep.set_publish_handler(
                        [session]
                                (MQTT_NS::optional<packet_id_t> packet_id,
                                 MQTT_NS::publish_options pubopts,
                                 MQTT_NS::buffer topic_name,
                                 MQTT_NS::buffer contents){
                            /* std::cout << "[server] publish received."
                                      << " dup: "    << pubopts.get_dup()
                                      << " qos: "    << pubopts.get_qos()
                                      << " retain: " << pubopts.get_retain() << std::endl;
                            if (packet_id)
                                std::cout << "[server] packet_id: " << *packet_id << std::endl;
                            std::cout << "[server] topic_name: " << topic_name << std::endl;
                            std::cout << "[server] contents: " << contents << std::endl; */

                            if(!mqtt_valid_topic(topic_name))
                                return false;

                            if(pubopts.get_retain() == MQTT_NS::retain::yes)
                                retained_map.insert_or_update(topic_name, std::make_pair(topic_name, contents));

                            std::map< session_ptr_t, MQTT_NS::qos > subscribers;
                            subs_map.find(std::string(topic_name.data(), topic_name.size()), [&topic_name, &contents, &pubopts, &subscribers](session_subscription_t const &r){
                                subscribers[r.session] = std::max(r.qos, subscribers[r.session]);
                            });

                            for(auto const &r: subscribers)
                                r.first->publish(topic_name, contents, std::min(r.second, pubopts.get_qos()) |  pubopts.get_retain());

                            return true;
                        });

                ep.set_subscribe_handler(
                        [session] (packet_id_t packet_id, std::vector<std::tuple<MQTT_NS::buffer, MQTT_NS::subscribe_options>> entries) {
                            std::cout << "[server]subscribe received. packet_id: " << packet_id << std::endl;
                            std::vector<MQTT_NS::suback_return_code> res;
                            res.reserve(entries.size());

                            auto sp = session->get_connection();

                            for (auto const& e : entries) {
                                MQTT_NS::buffer topic = std::get<0>(e);
                                MQTT_NS::qos qos_value = std::get<1>(e).get_qos();

                                if(mqtt_valid_subscription(topic)) {

                                    auto handle = session->get_subscription(topic);
                                    if(!handle.empty())
                                        subs_map.update(handle, session_subscription_t(session, qos_value));
                                    else
                                        session->add_subscription(topic, subs_map.insert(topic, session_subscription_t(session, qos_value)));

                                    res.emplace_back(MQTT_NS::qos_to_suback_return_code(qos_value));

                                    retained_map.find(topic, [&session, &topic, qos_value](std::pair<MQTT_NS::buffer, MQTT_NS::buffer> const &retained_message) {
                                        session->publish(retained_message.first, retained_message.second, qos_value);
                                    });
                                } else {
                                    res.emplace_back(MQTT_NS::suback_return_code::failure);
                                }

                            }

                            sp->suback(packet_id, res);
                            return true;
                        }
                );

                ep.set_unsubscribe_handler([session](packet_id_t packet_id, std::vector<MQTT_NS::buffer> topics) {
                            auto sp = session->get_connection();
                            std::cout << "[server]unsubscribe received. packet_id: " << packet_id << ", client id: " << session->client_id << std::endl;

                            for (auto const& topic : topics) {
                                auto handle = session->remove_subscription(topic);

                                subs_map.remove(handle, session_subscription_t(session));
                            }

                            sp->unsuback(packet_id);
                            return true;
                        }
                );
            }
    );

    s.listen();

    ioc.run();
}

If you really consider serializing the retained and published messages to disk, i would recommend using an embedded database such as: LevelDB orKyotoCabinet. And perform the actual writing on different thread then the network/mqtt handling.

Thanks you! I will take a look your codes.

If you really consider serializing the retained and published messages to disk, i would recommend using an embedded database such as: LevelDB orKyotoCabinet. And perform the actual writing on different thread then the network/mqtt handling.

Yes, I wrote examples on wiki https://github.com/redboltz/mqtt_cpp/wiki/Persistence
And my proprietary broker (not mqtt_cpp one) uses redis to store message.

Not sure how easy it would be to add this to the design now, that is something to consider for the future ?

test_broker.hpp is originally designed for testing for client code.
Now, it is becoming one of reference implementation of MQTT v5 broker.
Perhaps test_broker.hpp is moved to something like utility/simple_broker.hpp .

Tsumaranai monodesuga ( hope this is right thing to say :D )
It is perfect! つまらないものですが

Thank you for interested in Japanese :)

https://github.com/redboltz/mqtt_cpp/blob/a897c81932a0b363388b1bc53a56b8ab61175538/test/subscription_map.hpp#L390-L393
It seems that the function insert() calls insert_or_update(). Is this right?
I think that insert() should be failed if the same element exists.
The return value is similar to https://en.cppreference.com/w/cpp/container/set/insert
maybe std::pair<handle, bool>.

I need to know newly inserted or updated.
If subscription option retain_handling is send_only_new_subscription, then send retained message to the subscriber only if the subscription is new one.
See
https://github.com/redboltz/mqtt_cpp/blob/a897c81932a0b363388b1bc53a56b8ab61175538/test/test_broker.hpp#L981-L1012
https://github.com/redboltz/mqtt_cpp/blob/a897c81932a0b363388b1bc53a56b8ab61175538/test/test_broker.hpp#L1073-L1098

maybe std::pair<handle, bool>

I agree, but an optional is better in this case. Also, the insert in the single subscription map throws an exception if a duplicate store is detected. I will also change that to returning an optional.

https://github.com/redboltz/mqtt_cpp/blob/a897c81932a0b363388b1bc53a56b8ab61175538/test/subscription_map.hpp#L390-L393
It seems that the function insert() calls insert_or_update(). Is this right?
I think that insert() should be failed if the same element exists.
The return value is similar to https://en.cppreference.com/w/cpp/container/set/insert
maybe std::pair<handle, bool>.

I don't think it should fail when the element already exists. The c++ container set also does not really fail, it just states whether it inserted the element or it is returning the existing element. So actually the set insert is insert_or_update. In case of a single subscription map, it should fail.

Ok, I come up with a new question. What is the difference between single_subscription_map and multiple_subscription_map?
I think that it is similar to map and multi_map relationship.
I think that multiple_subscription_map::insert_or_update() is quite misleading.Because it always insert never update. Is that right?
I think that multiple_subscription_map::insert_or_update() should be removed.
How about multiple_subscription_map::update() ? I'm not sure the behavior.
single_subscription_map::insert_or_update()  doesn't exist. I think that it should exist.
What do you think?

I think that multiple_subscription_map::insert_or_update() is quite misleading.Because it always insert never update. Is that right?

yes, exactly, insert_or_update should be renamed to insert, same as in a set.

How about multiple_subscription_map::update() ? I'm not sure the behavior.

The update with the handle, inserts the value if the subscription already exists.
Not sure what to name this, or maybe just remove.

Thanks.
In multiple_subscription_map, I think that the following update is good:

remove handle insert(MQTT_NS::string_view subscription, Value const& value).
rename handle insert_or_update(MQTT_NS::string_view subscription, Value value) to handle insert(MQTT_NS::string_view subscription, Value value)
rename void update(handle h, Value value) to void insert(handle h, Value value)`.

If it is OK, I will update. And then I start implementing test_broker using #687  way.
How about single_subscription_map ?
BTW, I only use multiple_subscription_map in test_broker.hpp. So single_subscription_map is not so important for me.

@redboltz Please have a look at #688

The remove returns a boost::optional, the value that is returned is actually the value that you give when calling the function, and the optional indicates whether a value was actually removed or not. If i look at the std::set, the operation is called erase and not remove, and actually the result is a count of the number of items removed:
(2) size_type erase (const value_type& val);
For the value-based version (2), the function returns the number of elements erased.
Member type size_type is an unsigned integral type.
This actually makes more sense

Now I added the remove -> erase rename to #688, not sure if that was a good idea.

I will ask you rebase #688 from #687 later. I'm fixing #687,

https://github.com/redboltz/mqtt_cpp/pull/687/checks?check_run_id=1263724124
I stuck the error. It is caused by hash related issue required by subscription_map requirement .
Any ideas?

Any ideas?

hash_value is not implemented for boost::basic_string_view<char, std::char_traits, so I guess we need to give implementation of this function for hashing

Or include boost_string_view header?

It seems that boost::basic_string_view has it
https://www.boost.org/doc/libs/1_74_0/boost/utility/string_view.hpp

Yes, I noticed. Maybe include it explicitely ?

I found the reason.
hash_value function has been introduces since 1.69.0.
Maybe I can fix the problem.

Give implementation of hash_value function, boost version < 1.69

The unit test should include the checking subscription is removed when it is empty in multiple_subscription_map, when all the values are removed from a subscription, the subscription can be removed completely from the map. Also the reference counting should be checked in the unit test. In my test I included checking that when all subscriptions are removed, you only have the root element left: all entries are removed from the hash map, expect the root element

I think the reason was why i used a shared_ptr, is that the connections are not only indexed by the multiple_subscription_map, but by other keys in a multi index as well. You can not add the multiple_subscription_map to another multi_index structure, so I used a shared_ptr. In case a connection disconnects, i remove all the subscriptions from the multiple_subscription_map and this removes all shared_ptr's to the connection. Finally, the last shared_ptr is removed from the multi_index.

In my test broker, multiple subscriptions may point to the same session. Therefore I use a shared_ptr. The session itself (shared_ptr to session), is also attached to the lambda functions for handling incoming messages for a connection.

Thank you. In test_broker.hpp design, multiple con_sp_t are shared by sub_con_online, so sub_con_sp_t is a shared_ptr.
Each subscription has one sub_con_online element. So it doesn't need to be a shared_ptr.
One sub_con_online has one entry in subscription_map. So I choose reference_wrapper<sub_con_online> for subscription_map.

I merged #687. I renamed subscription_map.cpp to subscription_map_broker.cpp.
If you have a time, please add your tests for subscription_map as subscription_map.cpp.

I understand, but I would never use reference_wrapper. The pointer in reference_wrapper may become invalid. Cleanup of the subscription_map should be done before the referenced pointer is freed. This complicates cleaning up the data, in case of exceptions, or possibly when code is modified later an error is made. Usage of weak_ptr, reference_wrapper, raw pointers, etc... in almost all cases indicate a mistake in ownership of data.
Ownership of sub_con_online is shared between entries in subscription_map and mi_sub_con_online, therefore a shared_ptr makes sense. Removing entries from subscription_map and mi_sub_con_online is now independent on order of removal.
A language such as Rust would not allow such a construct, lifetime of both pointers may only partially overlap and one pointer may outlive lifetime of referenced object.
Hopefully you understand and appreciate my explanation.

I understand your design policy. But I don't choose it.
I know how dangling reference is dangerous.
In this case, it is not so difficult to ensure the lifetime.
Maybe sub_con_online destructor, if handle is not empty (perhaps this check is not required if adding subscription_map is done in the constructor parameter initializer), then erase the handle from subscription_map.
Anyway, I will implement broker by my way.

@kleunen , I found a problem.
When topic filter is overlapped, I got "Invalid handle was specified" exception on erase.
Here is a minimal code that reproduces the error.
"overlapped" means subscription topic filter are "a/b/c" and "a/b" relationship. They should be independent.
But in the directory metaphor, "a/b" is the parent directory of "a/b/c".
What do you think?
    using func_t = std::function<void()>;
    using value_t = std::shared_ptr<func_t>; // shared_ptr for '<' and hash
    using sm_t = multiple_subscription_map<value_t>;

    sm_t m;
    auto v1 = std::make_shared<func_t>([] { std::cout << "v1" << std::endl; });
    auto v2 = std::make_shared<func_t>([] { std::cout << "v2" << std::endl; });

    auto it_success1 = m.insert("a/b/c"_mb, v1);
    assert(it_success1.second);
    auto it_success2 = m.insert("a/b"_mb, v2);
    assert(it_success2.second);

    auto e1 = m.erase(it_success1.first, v1);
    std::cout << e1 << std::endl;
    auto e2 = m.erase(it_success2.first, v2); //  Invalid handle was specified is thrown here
    std::cout << e2 << std::endl;

I got m.dump().
after v1 is inserted
2 c 1 1
1 b 0 1
0 a 0 1
18446744073709551615  0 1

after v2 is inserted
2 c 1 1
1 b 1 1
0 a 0 1
18446744073709551615  0 1

after v1 is erased
18446744073709551615  0 1

I will have a look, that is not correct indeed

Thanks!
It happens not only subscription_map but also retained_topic_map.

I am a bit lost now what is the branch / PR i should work on ?

Now, subscription_map and retaind_topic_map are merged into the master.
If you fix the issue, then please create a new PR against master.
The problem is happened on my local environment. Not yet pushed.

But the _mb is to make the string multibyte ? I added a test for the subscription_map, but the _mb is not part of the main project ?

ah yes, we broke the insert.

_mb is postfix of mqtt::buffer.

But the _mb is to make the string multibyte ? I added a test for the subscription_map, but the _mb is not part of the main project ?

Now, broker oriented subscription_map test is in subscription_map_broker.cpp.
There is no pure subscription_map tests in the master. If you add it, it is very welcome. Please use the filename subscription_map.cpp for that.

Yes, I added the test, with your broken case. #558
and the fix

Sorry ? I'm not sure what's going on.
I have no plat to merge #558
The master has already have subscription_map and retainded_topic_map.
Please add the fix against the master and add tests.

Ok. I made mistake somewhere while pushing to github. Allow me to make proper PR

What about naming in retained_topic_map ? Should we keep insert_or_update, or make this insert also ?
It is actually an insert_or_update, but makes sense to use the same names

The current interface is
template<typename Value>
class retained_topic_map {

    retained_topic_map();

    // Insert a value at the specified subscription path
    void insert_or_update(MQTT_NS::string_view topic, Value const& value);

    // Find all stored topics that math the specified subscription
    void find(MQTT_NS::string_view subscription, std::function< void (Value const&) > const& callback) const;

    // Remove a stored value at the specified topic
    void remove(MQTT_NS::string_view topic);
;

I think that the following is good.
template<typename Value>
class retained_topic_map {

    retained_topic_map();

    // return inserted or updated object iterator (I'm not sure the type. Something like iterator type)
    // value as forwarding reference. User can move (or pass a temporary value) into the map
    template <typename V>
    iterator insert_or_update(MQTT_NS::string_view topic_name, V&& value);

    // Find all stored topics that math the specified subscription
    void find(MQTT_NS::string_view topic_filter, std::function< void (Value const&) > const& callback) const;

    // Remove a stored value at the specified topic
    // return value is 0 or 1. But std::size_t is good.
    // It is out of MQTT spec but if we would have multple_retained_topic_map, then std::size_t is consistent type
    std::size_t erase(MQTT_NS::string_view topic_name);
;

I added the unit test and fixed a number of bugs in the reference counting. Updating the interface i would like to do later. Have a look at PR #689

Thank you! I just added review comments.

    // return inserted or updated object iterator (I'm not sure the type. Something like iterator type)
    // value as forwarding reference. User can move (or pass a temporary value) into the map
    template <typename V>
    iterator insert_or_update(MQTT_NS::string_view topic_name, V&& value);

Yes, the retained_topic_map does not have the handles to topics like the subscription_map does. You can return a handle here, which is smaller to store than the complete topic_name.

I am not sure if it is really worthwile for retained_topic_map. For the subscription_map, you have to clear the subscriptions quite often, when the client disconnects. But the retained_topics are always retained, you do not have to store references to these topics in the broker.

Returning iterator by insert_or_update() is not important for test_broker implementation.
I have no problem with the current interface.
If it is easy and cheap operation, it's nice to have. But it isn't, just skip it.

The forwarding references are useful, but should we add that also to the erase operation ? This is now a Value const &, but an rvalue can not be passed to this function now.

If you add (or modify) forwarding reference to insert_or_update() it is nice.
It can reduce copy operation.
For erase, no value parameter is required.
I'm not sure what is the meaning of value parameter because single retained_topic_map has at most one value for one key (topic_name).

I'm not sure what is the meaning of value parameter because single retained_topic_map has at most one value for one key (topic_name).

single_value_map has no Value parameter in erase. Returned value can be either 1 or 0, a boolean could also have been used. But I guess the std::set also uses a size_t on erase:
class single_subscription_map {
    std::size_t erase(MQTT_NS::string_view subscription);
}

I'm not sure what is the meaning of value parameter because single retained_topic_map has at most one value for one key (topic_name).

erase in retained_topic_map does not have Value parameter, same for single_subscription_map. Return value (for both erase operations) is a size_t which is either 1 or 0. This is the same as for multiple_subscription_map and std::set. So this is OK, i would say
class retained_topic_map {
    std::size_t erase(MQTT_NS::string_view topic);
}

class single_subscription_map{
    std::size_t erase(MQTT_NS::string_view subscription);
    std::size_t erase(handle h);
}

I'm thinking that I use retained_topic_map for manage actual data struct retain, not view. It is very different from subscription_map usercase.
Subscribe related operations require multiple keys if I want to do the operations efficiently. As I commented before, when a connection that is SessionExpiryInterval:0 is disconnected, all subscription relates to the connection need to be unsubscribed.
It requires searching container by connection. subscription_map doesn't provide it. So I use multi_index for that and subsctiption_map is only for wildcard matching as view class.
But retained_topic_map is different. The operation is add/remove on publish received. If the contents is empty, remove, otherwise add. Find by topic filter happens when subscribe is received. So the all operation is covered by retained_topic_map efficiently.

And MQTT_NS::string_view subscription should be passed by copy right ?
So: MQTT_NS::string_view subscription
and not: MQTT_NS::string_view subscription const & ?

I think that

And MQTT_NS::string_view subscription should be passed by copy right ?

Yes.

Subscribe related operations require multiple keys if I want to do the operations efficiently. As I commented before, when a connection that is SessionExpiryInterval:0 is disconnected, all subscription relates to the connection need to be unsubscribed.
It requires searching container by connection. subscription_map doesn't provide it. So I use multi_index for that and subsctiption_map is only for wildcard matching as view class.

That is what the handle was envisioned for: fast removal of subscriptions. So for every Session you can store a list of all subscription handles that are in the subscription map. And when Session is removed, remove all the subscriptions by handle.

I got confused.
I think that as follows:
using sm_t = multiple_subscription_map<int>;
sm_t m;
int v1 = 1;
int v2 = 2;
auto [h1, r1] = m.insert("a/b", v1);
auto [h2, r2] = m.insert("a/b", v2);

// I think that the following two function causes the same result but
// handle version is more efficient (skip searching process)
m.erase(h1, v1);
m.erase("a/b", v1);

Is that right?

Yes,
m.erase(h1, v1); // Is more efficient
m.erase("a/b", v1); // Is less efficient

But also storing h1 is more efficient to store, than storing "a/b", memory size is smaller.

And one session can store multiple handles that is corresponding to subscription.
Then I can call as follows:
for (auto const& h : session.handles) {
    m.erase(h, ?);
}

And it is efficient. Is that right?

And it is efficient. Is that right?

Yes, exactly. And ? is a reference to the session itself.

I use handle for that purpose in test_broker.hpp to manage subscription.
But retained_topic_map, I don't use the handle version of erase(). (It is not provided though)
For implement test_broker retain mechanism, erase("a/b"); is good enough.
Because there is no other related containers.

So I requested adding handle return value for retained_topic_map::insert_or_update(), but I don't need it actually. Just for consistent interface. So it is low priority.

CI breaks now :) with perfect-forwarding

https://github.com/kleunen/mqtt_cpp/blob/sub_test_3/test/subscription_map.hpp#L439
V&& is forwarding reference not rvalue reference. So you need to use std::forward<V>(value) instead of std::move(value).

It requires searching container by connection

Yes, you can not search through the map based on value. Also, when storing the handle of a subscription, the complete path through the tree is stored. But actually only the leaf is required. You can trace back the parent. So some performance can still be gained for this operation.

Please consider the performance optimizations in #690, it removes a lot of temporaries when performing certain operations and makes the handle smaller (only the key of the leaf in the subscription tree).

Would be nice to have some public server for this broker as well, once the trie based wildcard support is in.

Regarding the retained topic, there is still an optimization that can be made. A map is now used, the entries are ordered. This is because for a wildcard (#/+), we need to find all topics that match under a certain child (a/#, search all childs of a). But in case we need to lookup a specific child (find a/b), we could use a hash map lookup. This would be possible if we use a multiindex for the retained topics. Use a hash for named lookups (like in the subcription map), and use a ordered_non_unique based on the id of the parent root for evaluating wildcards.

Would be nice to have some public server for this broker as well, once the trie based wildcard support is in. Like the mosquitto test server: Mosquitto Test

Yes, it is nice to have. But I don't know where is appropriate place.
I personally don't have a plan using my personal AWS (EC2 or container).
And now, MQTTv5 is not completely implemented. I'm implementing it. When all features are implemented, it is a time to publish.

Regarding the retained topic, there is still an optimization that can be made. A map is now used, the entries are ordered. This is because for a wildcard (#/+), we need to find all topics that match under a certain child (a/#, search all childs of a). But in case we need to lookup a specific child (find a/b), we could use a hash map lookup. This would be possible if we use a multiindex for the retained topics. Use a hash for named lookups (like in the subcription map), and use a ordered_non_unique based on the id of the parent root for evaluating wildcards.

Additional optimizations are very welcome!
I'm integrating trie based map into the broker. If the interface is not changed, it is better but I can accept some changes.

mqtt_cpp supports C++14 or later. When you use boost::multi_index, please don't use https://www.boost.org/doc/libs/1_74_0/libs/multi_index/doc/reference/key_extraction.html#key
It requires C++17.
I sometimes asked why you support C++14 ?
My policy is support one older version of the current up to date version.
Now, C++17 is up to date version. I know about C++20 but not all major compilers support it yet.
If most of major compilers support C++20, then I would update mqtt_cpp minimum standard requirement to C++17.

I will fix size first i think. That way you can add some debugging to the broker. Maybe publish to $sys

I implemented the calculation of the number of subscriptions / retained topics in #695. Also, the PR contains another fix for a reference counting issue in the retained topic map. Additional tests were added

Yes, it is nice to have. But I don't know where is appropriate place.
I personally don't have a plan using my personal AWS (EC2 or container).

I was thinking maybe I get a Pine64 Rock64 and host from home. I have 1Gbit glassfiber connection.

Would be nice to have some public server for this broker as well, once the trie based wildcard support is in. Like the mosquitto test server: Mosquitto Test

Yes, it is nice to have. But I don't know where is appropriate place.
I personally don't have a plan using my personal AWS (EC2 or container).
And now, MQTTv5 is not completely implemented. I'm implementing it. When all features are implemented, it is a time to publish.

Options: AWS free tier -- I do not know how this works but its worth considering.
Host on contributor connection -- risky if the connection is consumer grade, but viable if business class.
Create a digital ocean market place virtual machine template -- allows people to have a "one click" installation to try out themselves.
Partner with company that uses mqtt_cpp. -- if I were still at my previous job I would offer to do this, so there may be companies that use mqtt_cpp that would be willing to do this.

Concerns:
Need to implement bandwidth limiting and quotas to avoid abuse. This would include persisting statistic information.
Need to implement some kind of security policy modules if hosted by third party. E.g. support policies to allow limiting who can subscribe to which topics based on clientid / auth negotiation / group membership specified by some database or config file. This is important for a company so that they can feel like they can secure the broker against malicious use. Even in a "its just a demonstration server" situation, no company wants to host a service that has zero security controls.
Should convert broker from sync-only to async-only. Important for performance when you have >hundreds of clients.
Test_broker currently does not try to support multiple threads -- if it were being hosted for demonstration purposes, it should use #threads equal to the hardware its running on to demonstrate the high throughput its capable of.
Consider adding some hooks for operating the test broker in a cluster environment, like kubernates. This includes supporting health checking functions like "$healthcheck/kubernates" or whatever topic. It doesn't mean that you'd need to operating more than one instance of the test broker at a time, but just supporting the health checking logic that kubernates needs to understand if it needs to restart the application. Or migrate it to another higher capacity node.

Concerns:
Need to implement bandwidth limiting and quotas to avoid abuse. This would include persisting statistic information.

This can be handled at OS / Router level, you don't have to build this into the broker.

Need to implement some kind of security policy modules if hosted by third party. E.g. support policies to allow limiting who can subscribe to which topics based on clientid / auth negotiation / group membership specified by some database or config file. This is important for a company so that they can feel like they can secure the broker against malicious use. Even in a "its just a demonstration server" situation, no company wants to host a service that has zero security controls.

I use mosquitto now in production environment, indeed authorization and security is the biggest thing missing now.

Should convert broker from sync-only to async-only. Important for performance when you have >hundreds of clients.

Definitly required to have good performance.

Test_broker currently does not try to support multiple threads -- if it were being hosted for demonstration purposes, it should use #threads equal to the hardware its running on to demonstrate the high throughput its capable of.

This you don't have to do in the broker itself. When you can do is to use the SO_REUSE to provide load balancing over multiple broker processes. See:SO_REUSE
For example you can start 4 broker processes: broker[0], broker[1], broker[2], broker[3]
Now, with the SO_REUSE, the clients connect to the 4 different brokers. They subscribe to there respective tokens there.
Only thing is that when 1 client publishes a message, it will be received by 1 broker only, for example broker[0]. What you need is to have 1 republisher client, which connects to all 4 brokers and subscribes to all messages published. And this client republishes every message received on a broker on all other brokers.
You need to have 1 port which connects to all brokers in the cluster:
port 10000 -> broker[0], broker[1], broker[2], broker[3]
And other ports which connect to specific brokers:
port 10001 -> broker[0]
port 10002 -> broker[1]
port 10003 -> broker[2]
port 10004 -> broker[3]
For the republishing client to use. Not sure how to extend this to shared subscriptions, I guess shared subscriptions need to be republished in a round-robin fashion until one broker has handled the message.

Consider adding some hooks for operating the test broker in a cluster environment, like kubernates. This includes supporting health checking functions like "$healthcheck/kubernates" or whatever topic. It doesn't mean that you'd need to operating more than one instance of the test broker at a time, but just supporting the health checking logic that kubernates needs to understand if it needs to restart the application. Or migrate it to another higher capacity node.

Definitely useful

The PR with updated indices compiles and tests OK now on my CI, i can open new PR. Issue was the boost version check in the header was not OK

Regarding the hosting I am seriously considering upgrading my NAS and maybe providing the ability to host a docker image from my home. I have a readynas duo which is running for 10 years now, and needs upgrading anyway. I think the github build system can build a docker image, right ? This should be easy to host.
Regarding the authentication, I was thinking maybe the broker can provide authentication plugins by publishing authentication requests to a specialized topic. A client can then authenticate the request and give the broker a authenticated message back to the broker. Both the broker and the authentication client will have to use a shared secret key, to make sure the authentication client is authenticated to authorize other clients.
I think this is a pretty convenient approach. You can write an authentication client in PHP for example and retrieve the authentication from a database or a file. The broker still needs some basic authentication to allow for this client to access the broker.
Is this something interesting to work out ?

Bug #128 can be closed now right ?

Bug #128 can be closed now right ?

Ah, yes I just did it.


Address route and subpipeline for pipeline tranformation

Description
Address route and subpipeline for pipeline tranformation.

PipelineModel uses route and routes as alias, which needs to be addressed in the transformation.
Routes or route (defined as alias in Pipeline Model) will be always be transformed to route in json , as route will be primarily picked in pipelineModel when converting model to json or the other way round.

Subpipelines within one pipeline was addressed in the transformation.
Earlier, subpipelines were using the untransformed pipeline name. Made changes to make it pick the transformed pipeline name.

Check List

[x] New functionality includes testing.
[ ] New functionality has a documentation issue. Please link to it in this PR.

[ ] New functionality has javadoc added

[ ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Routes or route (defined as alias in Pipeline Model) will be always be transformed to route in json

We should always transform it to routes.