Ensure the arm templates include a clientID or some guid that can be used for telemetry

We need to make sure the arm templates use the most recent mechanisms for identifiying themselves as the source of a provision action.

@benbrown do you mean this, we do have this in provision script, also in composer provision flow. We can view the provision data from the armprod.

@benbrown do you mean this, we do have this in provision script, also in composer provision flow. We can view the provision data from the armprod.

But I found the armprod only save the data about 1~2 month, the old data are all gone... I am not sure if we can get it back.

But I found the armprod only save the data about 1~2 month, the old data are all gone... I am not sure if we can get it back.

https://github.com/microsoft/BotFramework-Composer/blob/5af3f50ef41e3ef471771deb141437b3c3ae6e92/extensions/samples/assets/shared/scripts/DeploymentTemplates/template-with-preexisting-rg.json#L150
I just found that this function has been commented out, we should get it back to collect the data.
https://github.com/microsoft/BotFramework-Composer/blob/d79ef62f1c36242260c1db29f607c87482cadce6/extensions/azurePublish/src/node/provision.ts#L371

https://github.com/microsoft/BotFramework-Composer/blob/5af3f50ef41e3ef471771deb141437b3c3ae6e92/extensions/samples/assets/shared/scripts/DeploymentTemplates/template-with-preexisting-rg.json#L150
I just found that this function has been commented out, we should get it back to collect the data.
https://github.com/microsoft/BotFramework-Composer/blob/d79ef62f1c36242260c1db29f607c87482cadce6/extensions/azurePublish/src/node/provision.ts#L371

Thanks to everyone who contributed to this effort!
Our PM team confirms that this data is what we need for now.


Portal Site Creation Issue

[2017-08-09T04:06:06+00:00] INFO: Creating portal site...
[2017-08-09T04:22:57+00:00] INFO: Processing arcgis_enterprise_portal[Configure HTTPS] action configure_https (arcgis-enterprise::portal line 133)
[2017-08-09T04:22:57+00:00] INFO: Processing arcgis_enterprise_webadaptor[Unpack ArcGIS Web Adaptor] action unpack (arcgis-enterprise::portal_wa line 20)
[2017-08-09T04:22:57+00:00] INFO: Processing arcgis_enterprise_webadaptor[Install Web Adaptor for Portal] action install (arcgis-enterprise::portal_wa line 38)
[2017-08-09T04:22:57+00:00] INFO: Processing arcgis_enterprise_webadaptor[Deploy Web Adaptor for Portal] action deploy (arcgis-enterprise::portal_wa line 55)
[2017-08-09T04:22:57+00:00] INFO: Processing arcgis_enterprise_webadaptor[Configure Web Adaptor with Portal] action configure_with_portal (arcgis-enterprise::portal_wa line 65)
[2017-08-09T04:23:57+00:00] ERROR: Failed to configure Web Adaptor with Portal for ArcGIS. Expected process to exit with [0], but received '1'
---- Begin output of "C:\Program Files (x86)\Common Files\ArcGIS\WebAdaptor\IIS\Tools\ConfigureWebAdaptor.exe" /m portal /w "https://myFQDN/portal/webadaptor" /g "https://myserverIP:7443" /u "myadminname" /p "censored" ----
STDOUT: Configuring Web adaptor...
Unable to register the Portal with the Web Adaptor. Please make sure that that the Portal is running and that the account specified has administrative privileges.
STDERR:
---- End output of "C:\Program Files (x86)\Common Files\ArcGIS\WebAdaptor\IIS\Tools\ConfigureWebAdaptor.exe" /m portal /w "https://myFQDN/portal/webadaptor" /g "https://myserverIP:7443" /u "myadminname" /p "censored" ---- 4 tries remaining.
After this the stack fails to complete and the primary server stops. The admin account and creds are correct. The previous issues I had in v2.3.1 I was able to resolve, however in v3.1.0 I'm having issues with portal site configuration.
Manually running the command in cmd, I got an error that PortalURL does not exist!!!!!!! Please assist. If I manually enter https:////portal/webadaptor or the IPaddress:7443, it loads the ArcGIS portal site just fine and requires configuration[creating or joining site]. This is why i'm not sure why I get the error when it's run via cloudformation stack.

running into a similar issue.  What resolved this problem for you?


Feature: Standalone WinRT server for background operations

What feature or improvement do you think would benefit Files?
We initially chose Win32 Shell APIs for file operations because we wanted the file operations to continue even after Files was closed, but Win32 Shell APIs are designed for automation, not for the general use by a file explorer. And it comes with its own limitations (e.g. slow, no built-in transfer speed calculation and etc.).
We should consider implementing file operations as an out-of-process WinRT exe server, so that we no longer need to rely on the Win32 Shell APIs, neither facing the performance and maintainability issues due to serialization and inter-process communication. By using WinRT we can expose the APIs with WinRT types directly (such as IAsyncOperation<T> and Stream), and call them directly from another process.
Requirements
It was possible only with C++/WinRT, and CsWinRT still doesn't have built-in support for authoring OOP WinRT exe server for now (see the original feature request in CsWinRT: https://github.com/microsoft/CsWinRT/issues/1177).
But recently a project Shmuelie.WinRTServer has this implemented, so I think we can start to investigate it.
Files Version
Planning for post 3.0
Windows Version
10.0.22631.2506
Comments
No response

You mean implementing file ops using Win32 API? E.g CopyFileEx, MoveFileWithProgress, etc?

Yes. Implementing file ops using Win32 API in an external process and expose those APIs through WinRT ABI in an external process. This is similar with what we did before in fullTrustedProcess but not the same. By authoring a WinRT exe server, we don't need to serialize and dispath messages by ourselves, instead we cancan avoid the overhead of json serialization

Can a WinRT exe server run as administrator?

Can a WinRT exe server run as administrator? (without the main app running as admin)

We may not be able to activate an elevated WinRT type from a non-elevated process.

Will this help us support third party handlers? #7518

The investigation and experiment have done.
Note this is not a Code Quality work but a Feature work (which allows us to have a standalone background server for long-running tasks)

Why not simply just run the regular executable but without a window open? It will be much simpler. There are not that many resources about creating an out-of-process WinRT server.

Why not simply just run the regular executable but without a window open? It will be much simpler. There are not that many resources about creating an out-of-process WinRT server.

Because then you will need to handle all serialization stuff and message dispatch by yourself, which results in major maintainability regression.

Because then you will need to handle all serialization stuff and message dispatching by yourself,
No I mean just do it in the same process as the main app. Just hide the window.

No I mean just do it in the same process as the main app. Just hide the window.

This will still face the concurrency issue. We want all operations to happen in a single process to allow us to update to LiteDB 5.

The investigation and experiment have done.  Note this is not a Code Quality work but a Feature work (which allows us to have a standalone background server for long-running tasks)

That's good to hear, keep us updated!

We want all operations to happen in a single process
I did mean the same process as the app, meaning hitting X on the window will only hide it and the operations will continue to run.

This will still face the concurrency issue. We want all operations to happen in a single process to allow us to update to LiteDB 5.

What if we are going to make the app single instantiated? When user tries to open another window, Program class can redirect and App class can create another Windows class within the same App class while it would make things much complicated such as the DI container and MainWindow.Instance property. Tracked in #12689. In this case, oop WinRT server wouldn't be required.
Speaking of this, how we can cope with #7275? IFileOperation runs on STA. I wonder how File Explorer managed to run in parallel.
@hez2010 What do you think?


Error Installing through Pip on Python3.7 on Linux Ubuntu 18.04

I'm unable to install this on my local Linux Ubuntu 18.04 running Python3.7.
My current install process / what I have tried:
Try installing some possible missing dependencies.
Create a fresh new virtual environment for Python3.7
sudo apt-get install -yqq --noinstall-recommends apt-utils gcc build-essential
Try upgrading these packages
python3.7 -m pip install --upgrade pip setuptools wheel
Try installing the package
python3.7 -m pip install --no-cache-dir pyxDamerauLevenshtein
Note: These commands worked on a Docker image using Python3.7:slim-buster as the base image. I have also tried this
on a few other machines running 18.04
Current Output:
Defaulting to user installation because normal site-packages is not writeable
Collecting pyxDamerauLevenshtein
  Downloading pyxDamerauLevenshtein-1.6.1.tar.gz (55 kB)
     |████████████████████████████████| 55 kB 2.9 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.7 /home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp_901baqu
       cwd: /tmp/pip-install-wnfgf6t1/pyxdameraulevenshtein_c6df6562c7e04fd6a22642aacd88bc26
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 99, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 /home/erb13020/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp_901baqu Check the logs for full command output.

Any hints?

Have you tried installing to a local virtual environment rather than the system root environment? That is, something like this:
sudo apt install python3-venv
python3 -m venv /path/to/env
source /path/to/env
pip install --upgrade pip setuptools wheel
pip install numpy
pip install pyxDamerauLevenshtein

What does that get you?

Are you still experiencing this issue? I've released a few new versions in the meantime, and you might try that as well.

I'm going to go ahead and close this issue since I'm not receiving any feedback.


Add requests session support

Hi,
With multiple requests it's very useful to have one connection per host as it saves time significantly when a Vault host is far away.

Thanks for the PR! Since one of my recent commits introduced a merge conflict for this PR, I have implemented this functionality in e17b4bf616e8abb7699cde1d3949865c296f9e2d


add save and copy buttons

This adds two buttons: "Copy to clipboard" and "Save as CSV..."
Closes #1

Awesome! Thanks, @haesleinhuepf!


Support CTAS(Create Table As Select ~) statement

🚧 This PR is a draft yet
Support CTAS(Create Table As Select ~) statement(#404)

[x] Add a test case: src/tests/alter/create_table.rs
[x] Add source to translate if there is query statements: src/translate/mod.rs
[x] Get source from ast: src/ast/mod.rs
[x] Pass source to executor: src/executor/execute.rs
[ ] ⏳ Implement CTAS

[x] Get source table's schema
[x] Create new table with the source schema
[x] Insert data from source to new table
[x] Clear redundant and repetitive code
[ ] Should we add payload?

[x] Add more test cases

Codecov Report

Merging #414 (6c6a02b) into main (6add0fa) will decrease coverage by 0.10%.
The diff coverage is 74.07%.

@@            Coverage Diff             @@
##             main     #414      +/-   ##
==========================================
- Coverage   91.01%   90.90%   -0.11%
==========================================
  Files         144      144
  Lines        8432     8478      +46
==========================================
+ Hits         7674     7707      +33
- Misses        758      771      +13

Impacted Files
Coverage Δ

src/executor/execute.rs
67.87% <0.00%> (-0.42%)
:arrow_down:

src/executor/alter/table.rs
78.66% <72.09%> (-11.08%)
:arrow_down:

src/tests/alter/create_table.rs
100.00% <100.00%> (ø)

src/translate/mod.rs
98.21% <100.00%> (+0.10%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6add0fa...6c6a02b. Read the comment docs.


AudioSourceReader could not be found

I'm trying to use AudioSourceReader in a UWP app:

I installed the latest Nuget https://www.nuget.org/packages/UWPAudioVisualizer/1.0.38
I added using AudioVisualizer; to the .cs
But I still get this error:

What am I doing wrong?
Thanks
P.s.
I was able to use: AnalogVUMeter, DiscreteVUBar, SpectrumVisualizer without problems

@clarkezone @tonuv any idea? thanks

I Solved downloading and compiling the library by myself because, I also made few updates and fixes, here there is my PR
https://github.com/clarkezone/audiovisualizer/pull/78

I will have a look.

Hi @clarkezone and @tonuv any news regarding this issue? thanks


[MongoDB] BUG: streamEventsByRevision omits first event and repeats last event.

Hi. I found the bug on the mongoDB implementation method streamEventsByRevision.
If I call streamEventsByRevision like this
eventStore.streamEventsByRevision('28e7fc75-869b-486e-9482-f3230f4375b4',  -1,  -1);

It returns stream with data
{ _id: '5f23b2a84cb94b4294d0c9280',
  streamId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregateId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregate: 'User',
  context: '__default',
  streamRevision: 1,
  commitId: '5f23b2a84cb94b4294d0c928',
  commitSequence: 0,
  commitStamp: 2020-07-31T05:56:56.707Z,
  payload:
   { type: 'BALANCE_ADDED',
     data: '{"addedAmount":50}',
     issuer: '',
     eventId: '' },
  position: null,
  id: '5f23b2a84cb94b4294d0c9280',
  restInCommitStream: 0,
  dispatched: false }
{ _id: '5f23b2aa4cb94b4294d0c9290',
  streamId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregateId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregate: 'User',
  context: '__default',
  streamRevision: 2,
  commitId: '5f23b2aa4cb94b4294d0c929',
  commitSequence: 0,
  commitStamp: 2020-07-31T05:56:58.235Z,
  payload:
   { type: 'BALANCE_ADDED',
     data: '{"addedAmount":50}',
     issuer: '',
     eventId: '' },
  position: null,
  id: '5f23b2aa4cb94b4294d0c9290',
  restInCommitStream: 0,
  dispatched: false }
{ _id: '5f23bd0fa78f0c63007711ee0',
  streamId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregateId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregate: 'User',
  context: '__default',
  streamRevision: 3,
  commitId: '5f23bd0fa78f0c63007711ee',
  commitSequence: 0,
  commitStamp: 2020-07-31T06:41:19.332Z,
  payload:
   { type: 'BALANCE_ADDED',
     data: '{"addedAmount":50}',
     issuer: '',
     eventId: '' },
  position: null,
  id: '5f23bd0fa78f0c63007711ee0',
  restInCommitStream: 0,
  dispatched: false }
{ _id: '5f23bd0fa78f0c63007711ee0',
  streamId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregateId: '28e7fc75-869b-486e-9482-f3230f4375b4',
  aggregate: 'User',
  context: '__default',
  streamRevision: 3,
  commitId: '5f23bd0fa78f0c63007711ee',
  commitSequence: 0,
  commitStamp: 2020-07-31T06:41:19.332Z,
  payload:
   { type: 'BALANCE_ADDED',
     data: '{"addedAmount":50}',
     issuer: '',
     eventId: '' },
  position: null,
  id: '5f23bd0fa78f0c63007711ee0',
  restInCommitStream: 0,
  dispatched: false }

As you can see first event has omitted and last event has duplicated.
The test script scenario couldn't catch the issue as it only checks if length of returned events is appropriate..
I think I can create a PR for quick fixing this bug. Looking into it rn.

fixed in v1.15.5


Implement excision

See http://docs.datomic.com/excision.html.

Requirements/description:
Excision cannot remove vocabulary, the bootstrap schema, or any other built-ins. This includes the :db/excise attributes themselves.
Excision records datoms that define the excision itself, and additionally removes the affected datoms from history. Indeed, recording these datoms is how one triggers an excision, though I'm sure we'll wrap this in a nice API.
Excision targets either an entity or an attribute. (If you want to treat an attribute as an entity, you need to specify attributes to excise.)
If an entity is targeted, all inbound and outbound datoms are removed, as are any component entities linked by an outbound datom.
An optional set of attributes can be specified to filter the set of affected datoms.
An optional timestamp or transaction ID filter can be specified, similarly.
If an attribute is targeted, it affect all entities, subject to filtering as above.
It's important that excision is deterministic: for a given database state, a given excision input should always remove the exact same set of datoms.

Implementation choices:
This is likely to hook into the same spot in the transactor that picks up schema changes:
        if tx_might_update_metadata {
            // Extract changes to metadata from the store.
            let metadata_assertions = self.store.committed_metadata_assertions(self.tx_id)?;

I suggest that at that point we synchronously compute an 'excision operation' that can be persisted in the database (ensuring that all excisions eventually and atomically take place), in a separate table, and then process that table after the transact commits. This is inline with Datomic's background behavior. We can expose a synchronous 'purge' call that blocks until all pending excisions are processed.
Excision will mutate the datoms table, the log, potentially the fulltext values table (Datomic doesn't support excision of fulltext, but I think we should), and update the cache. It'll have no interaction with syncing except that certain sync operations — particularly merges — should only proceed against a store with no pending excisions, for obvious determinism reasons.

It'll have no interaction with syncing except that certain sync operations — particularly merges — should only proceed against a store with no pending excisions, for obvious determinism reasons.

I started thinking about this a little after our conversation yesterday, @rnewman, and it seems like excision should interact with syncing.  Suppose that excision leaves data on the server: then a user in possession of their credentials could be compelled to provide them, and the data on the server could be recovered.  That's a different guarantee than we provide when deleting data right now, and it's a weaker guarantee.
Server-side excision would require some assistance from the client, and would have a very different work-load.  It might look like a bulk replacement of transactions in-place, rather than the append-only workload we're currently expecting.
Thoughts?

… it seems like excision should interact with syncing.

Yes, in ways that are already documented elsewhere — in the above I meant that local asynchronous excision markers don't really interact with local sync code.
Excision and remote timelines are naturally related: any rewritten transaction chunks need to be uploaded, their old transaction entries rewritten to point to the new chunks, and the old chunks flagged for deletion or garbage collection.
This is append-only for data chunks (modulo flagging for cleanup), and read-write for the transaction metadata entries themselves.
See "Rewriting history" in my protocol outline.


the rule 210710 is fired but there is no audit log for it

Hi,
based on modsecurity debug logs, the rule 210710 is fired by executing a curl command that contains a malicious content-type but related audit log is not created.
the curl command is as follows:
curl -X POST -H 'Content-Type: application/hal+json;' http://example.com

ModSecurity version (and connector): [ModSecurity v3.0.6 with nginx-connector (2497e6ac654d0b117b9534aa735b757c6b11c84f)]
WebServer: [nginx-1.20.2]
OS (and distro): [ Linux, ubuntu 18.04.5x]

Hello @ssh81 ,
You haven't provided very much information to go on.
For example, what is "rule 210710"? Can you provide the relevant debug log output (at level 9)?
When you say "related audit log is not created"? Do you mean that the audit log file is never created at all (for any rule)? Do you mean that you get output to the audit log file written for some rules, but not for 210710?

Hello,
the rule number 210710 protects against malicious content type. the debug log out put at level 9 is as follows:
(Rule: 210710) Executing operator "Rx" with param "^(?:GET|HEAD|PROPFIND|OPTIONS)$" against REQUEST_METHOD.
[1651467229] [/] [9] Target value: "POST" (Variable: REQUEST_METHOD)
[1651467229] [/] [9] Matched vars updated.
[1651467229] [/] [4] Rule returned 1.
[1651467229] [/] [4] Executing chained rule.
[1651467229] [/] [4] (Rule: 0) Executing operator "Rx" with param "^([^;\s]+)" against REQUEST_HEADERS:Content-Type.
[1651467229] [/] [9] Target value: "application/hal+json;" (Variable: REQUEST_HEADERS:Content-Type)
[1651467229] [/] [7] Added regex subexpression TX.0: application/hal+json
[1651467229] [/] [7] Added regex subexpression TX.1: application/hal+json
[1651467229] [/] [9] Matched vars updated.
[1651467229] [/] [4] Rule returned 1.
[1651467229] [/] [4] Executing chained rule.
[1651467229] [/] [4] (Rule: 0) Executing operator "Within" with param "application/octet-stream|application/x-amf|application/xml|application/json|application/x-www-form-urlencoded|application/soap+xml|text/html|text/plain|text/xml|multipart/form-data" Was: "" against TX:0.
[1651467229] [/] [9] Target value: "application/hal+json" (Variable: TX:0)
[1651467229] [/] [9] Matched vars updated.
[1651467229] [/] [4] Running [independent] (non-disruptive) action: setvar
[1651467229] [/] [8] Saving variable: TX:anomaly_score with value: 12
[1651467229] [/] [4] Rule returned 1.
[1651467229] [/] [9] Running action: log
[1651467229] [/] [9] Saving transaction to logs
[1651467229] [/] [9] Running action: ctl
[1651467229] [/] [9] Running action: log
[1651467229] [/] [9] Saving transaction to logs
[1651467229] [/] [9] Running action: log
[1651467229] [/] [9] Saving transaction to logs
[1651467229] [/] [4] Running (non-disruptive) action: tag
[1651467229] [/] [9] Rule tag: CWAF
[1651467229] [/] [4] Running (non-disruptive) action: tag
[1651467229] [/] [9] Rule tag: HTTP
[1651467229] [/] [9] This rule severity is: 2 current transaction is: 2
[1651467229] [/] [9] Saving msg: Request content type is not allowed by policy. Please update file userdata_wl_content_type.||||2
[1651467229] [/] [4] Running (disruptive)     action: pass.
[1651467229] [/] [8] Running action pass
about audit log file, I mean I get output to the audit log file written for some rules, but not for 210710.
I updated modsecuriy last December. (SHA1: 52958fa73c04ad6bb924497fd3d767f6e5e59411)

Hi @ssh81 ,
I do not see an obvious explanation in the log excerpt that you have provided.
If your rule set is from a 3rd-party provider you may wish to inquire with them if this is an expected outcome for "rule 210710".
However, some thing you can try and/or look for:

check the debug log for the logging phase, e.g.:

... [4] Starting phase LOGGING. (SecRules 5)
... [9] This phase consists of 0 rule(s).
... [8] Checking if this request is suitable to be saved as an audit log.
... [8] Checking if this request is relevant to be part of the audit logs.
... [5] Saving this request as part of the audit logs.
... [8] Request was relevant to be saved. Parts: 6006

Note, if instead of 'was relevant to be saved', you see something like 'is not interesting to audit logs'.

I'm assuming you're using SecAuditEngine RelevantOnly. Try experimentally changing the setting to 'On' and see if your rule now shows up in the audit log.

Look in the full debug log for the transaction for occurrences of 'Running action: noauditlog'

Try explicitly adding 'auditlog' as one of the actions for 210710

Beyond that I think I'd need to see the full rule text for all rules triggered during that transaction, as well as for any SecAction directives, and any other logging-relevant configuration -- and the full debug log for the transaction.

Anything further on this?


the email images you added

http://www.sitepoint.com/how-to-code-html-email-newsletters/

Oh
Thanks .-. lmao
WIll you be coming to help as a mentor?


Bump lang: org

https://github.com/org-roam/org-roam/tree/fd73da94107dd3069379a9564be9ccf5712db084 -> https://github.com/org-roam/org-roam/commit/c59d6c4f7c917cf8580df2797f331c53e404fdbe

Thanks for the PR! I've bumped org-journal (and the rest of the module) in 27a3d8b because some updates to our org module were necessary after updating org-journal, and this PR's bump commit didn't precisely follow the commit message format for bump templates (no worries, I haven't documented it anywhere yet). Thanks for bringing it to my attention!

Thanks @hlissner  for the update, and sorry for messing it up with the PR format. I was a little confused in how to write the commit's description.