Exclude commons-codec dep from httpclient.  WildFly Core doesn't ship…

… commons-codec, but full WildFly does. I am seeing security scanner warns about bugs in the version transitively depended on by httpclient, which is older than the unaffected version full WildFly uses. I don't want to put commons-codec under WildFly Core's dependency management, as that just makes it a hassle when full WildFly wants to upgrade.  So let's see if just excluding it works.
Note that core has some test fixtures that use httpclient and those uses may need the transitive commons-code dep, in which case this commit will fail CI and we'll need a different solution.
Note: If this passes CI I'll file a JIRA and update the commit message and PR title.

Nope; doesn't work; some tests involving BASIC auth use commons-codec Base64.

@bstansberry Feel free to raise an issue for us on the last point, should be fairly easy to move - we have our own Base64 handling in Elytron we can switch to.

@darranl Would that be a WFCORE or an Elytron issue?

@bstansberry from your comment it sounds like the issue is tests in WildFly Core using it for Base64 handling so should just be a WildFly Core issue needed, I can have a look at the test results here and see what is needed.

Retest this please

@bstansberry my PR doesn't contain your changes, if you want I can quickly take your commit and convert to a real PR so we can get it in.

This PR has been replaced by https://github.com/wildfly/wildfly-core/pull/4133


Partition table support on SD card (IDFGH-12063)

Is your feature request related to a problem?
Hi all,
Currently for reading and writing SD card with filesystems, we either can use FAT filesystem which is up to 32GB, or use LittleFS which is up to 2GB. But modern SD cards are often bigger than that. In order to make the full use of the whole SD card, I'd suggest adding support for partition table.
cc @adokitkat
Describe the solution you'd like.

Using esp_paritition is possible, but it needs quite a bit hacks. The size and address fields in esp_partition are all uint32_t which is limited to 4GB. I guess Espressif initially only consider using this library within their SPI flash scope, not for something bigger like SD cards.
Porting MBR partition support is also doable, and I think it's probably the most useful option, so then SD card is accessible on a normal computer with a SD card reader.
It would be great to support GPT (GUID partition table), but AFAIK it's harder and it's heavier for RAM consumption when parsing

Describe alternatives you've considered.
N/A
Additional context.
Here's an existing discussion: https://github.com/joltwallet/esp_littlefs/issues/173

Hi. Just a note:
If I understand it correctly, the 32GB limit of a FAT32 filesystem is only a fake limit pushed by Windows. FAT32 should actually support 16TB (and I read most operating systems permit up to 2TB). You can use cmd commands or a custom software to format 32GB+ SD cards on a PC and you should be able to use the format function in ESP-IDF as well (although I would disable the Task WDT in idf.py menuconfig so it won't trigger mid-formatting and resets the ESP - on ESP it will take some time with a such large capacity).

If I understand it correctly, the 32GB limit of a FAT32 filesystem is only a fake limit pushed by Windows. FAT32 should actually support up to 16TB (and I read most operating systems permit up to 2TB).

Ah yes... I forgot this matter! Indeed it's 16TB if sector size is 4KB.
But meanwhile FAT32 is a bad choice for embedded stuff. The partition may be corrupted if there's a power loss during write operations. LittleFS is a better choice but it's limited to 2GB. Thus I think partitioning is a better option.

Yes it kind of is... We want to address it and are developing a journaling feature for FAT filesystem as a separate component, however I don't know when it will be ready. That's why I would like to encourage users to use LittleFS, it's just more modern and optimized for embedded devices. Raising the 2GB limit to two digit number would be great though.
Native MBR partition table support could be worth to consider as well for more flexibility, but right we use f_mkfs from FATFS library which creates MBR/GPT for SD cards.

We want to address it and are developing a journaling feature for FAT filesystem as a separate component

Hmm yea, I just had a look and I just realised Keil MDK has that.

That's why I would like to encourage users to use LittleFS, it's just more modern and optimized for embedded devices.
Raising the 2GB limit to two digit number would be great though.

My colleagues and I read LittleFS's doc and code earlier and we think it might need quite a bit hack to make it bigger than 2GB. But currently for our use cases (mostly logging & statistic stuff) 2GB is enough anyway.

Native MBR partition table support could be worth to consider as well for more flexibility, but right we use f_mkfs from FATFS library which creates MBR/GPT for SD cards.

Yes indeed I think MBR/GPT probably is the right way to go. This could be particularly helpful for HMI devices or surveillance cameras with ESP32-P4, where large storage is necessary for dealing with multimedia content. Users can do something like, a few small-ish LittleFS partitions for the microcontroller to write logs, and one or two bigger FAT partitions for storing videos.


Training Data Memory?

Hi everyone
I just started using chatterbot and played around by importing different corpus, as well as adding manual conversations. It seems there is memory from previous conversations. For example, if I do training on the ubuntu database, then redotraining on the English corpus, it seems to have memory of responses from ubuntu. This is creating some very unusual responses

Hi @kail518, your observation is correct. Knowledge is retained by the chat bot each time it is trained.
Methods exist to remove all learned statements from the chat bot if you wish to do so. You can do:
my_chatbot.storage.drop()


Update from develop

Inludes update from develop. What is NOT included is the API test page created in https://github.com/OpenConext/OpenConext-engineblock/commit/02a12bc134955fe0137df2869c2f29efa3f9b63e - the session bug was not something we had and thus does not need to be resolved.
If the API test page is a requirement (I'd advise against it), this needs to be added.
Functional tests pass
behat-travis:
     [exec] ...................................................................... 70
     [exec] ...................................................................... 140
     [exec] ...................................................................... 210
     [exec] ...................................................................... 280
     [exec] ...................................................................... 350
     [exec] ...................................................................... 420
     [exec] ...................................................................... 490
     [exec] ...................................................................... 560
     [exec] ...................................................................... 630
     [exec] ...................................................................... 700
     [exec] ...................................................................... 770
     [exec] ...................................................................... 840
     [exec] ...................................................................... 910
     [exec] ...................................................................... 980
     [exec] ...................................................................... 1050
     [exec] ...........
     [exec]
     [exec] 38 scenarios (38 passed)
     [exec] 1061 steps (1061 passed)
     [exec] 2m41.359s

@quartje @thijskh @arothuis : all conflicts resolved, all tests are green.
After having decided on a course of action with respect to the API test page, this can be merged and 5.0.0-alpha1 can be tagged for deployment, being equal in feature completeness to develop.
the use of bin/makeRelease.sh has not been verified yet, but I imagine that should not be much trouble (even to do manually should it be needed).

I don't understand the problem with the API test page and why it can't be merged. I'd like to understand it.
Nonetheless it's not crucial so if it currently impedes progress we can continue without it for now.

It is not a problem of merging it (it has been merged), however the file modified (the controller) no longer exists in the 5.x range. Furthermore the template has not been added as part of the theme, but rather as a separate page and would have been removed when applying the theme. Both the controller code and template would thus have to be recreated in the 5.x range code.
This recreation is also not an issue in and of itself, but I was wondering what the use of a test page for an API that should not be publicly accessible is. Also, it seems that either it does nothing but check if your basic auth is correct, or actually process the posted configuration (if I understand it correctly) which to me would seem rather unsafe.
If the functionality is required I would suggest adding it, but with a feature toggle that by default is off so that it has to be a conscious choice to enable this page.
That said it could well be I misunderstood the functionality, but I opted for "rather safe then sorry" as it seemed unsafe to me and adding the functionality at a later point is trivial.

The page is intended so haproxy has something to do a healthcheck against. It should basically just respond with a 200 so we now the machine in question is up. It could do more but we need at least something that can return a 200 on GET or HEAD.

That is already provided in the heartbeat controller
(src/OpenConext/EngineBlockBundle/Controller/Api/HeartbeatController), this
would be a test page for metadata push api if I read it correctly?
On Thu, Jan 28, 2016, 08:46 Thijs Kinkhorst<EMAIL_ADDRESS>wrote:

The page is intended so haproxy has something to do a healthcheck against.
It should basically just respond with a 200 so we now the machine in
question is up. It could do more but we need at least something that can
return a 200 on GET or HEAD.
—
Reply to this email directly or view it on GitHub
https://github.com/OpenConext/OpenConext-engineblock/pull/259#issuecomment-176037560
.

Right. I don't think the test page is currently essential.

:+1:, will merge and tag.


Bug: rules_ios lldb issue with generated Swift headers

Description
lldb has problems with visibility of generated header defined @objc values for use in Objective-C headers. This can be sidestepped by running a RegEx replacement of:
-ivfsoverlay\$\(CURRENT_EXECUTION_ROOT\)/([a-zA-Z\-/_0-9\.]*)/([a-z]*)_swift_vfs\.yaml

With:
 -ivfsoverlay$(CURRENT_EXECUTION_ROOT)/$1/$2_swift_vfs.yaml - ivfsoverlay$(CURRENT_EXECUTION_ROOT)/$1/$2_vfs.yaml

In the examples/rules_ios/rules_ios.xcodeproj/bazel/Debug-swift_debug_settings.py file seemingly indicating an issue with picking up the ivfsoverlay copts that aren't present in the swift_library target in the rules_ios target cluster.
An example vfs copt is //iOSApp/Source/Utils:Utils_vfs that's present on //iOSApp/Source/Utils:Utils_objc:
$ bazel query --output=build //iOSApp/Source/Utils:Utils_objc
# /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/iOSApp/Source/Utils/BUILD:6:26
objc_library(
  name = "Utils_objc",
  tags = ["manual"],
  generator_name = "Utils",
  generator_function = "apple_framework",
  generator_location = "iOSApp/Source/Utils/BUILD:6:26",
  testonly = False,
  features = [],
  data = ["//iOSApp/Source/Utils:Utils_wrapped_resources_filegroup"],
  srcs = ["//iOSApp/Source/Utils:Foo.m"],
  non_arc_srcs = [],
  pch = "@build_bazel_rules_ios//rules/library:common.pch",
  defines = [],
  module_map = "//iOSApp/Source/Utils:Utils-modulemap/Utils.modulemap",
  hdrs = ["//iOSApp/Source/Utils:Foo.h", "//iOSApp/Source/Utils:Utils.h", "//iOSApp/Source/Utils:Utils-modulemap/Utils-umbrella.h", "//iOSApp/Source/Utils:Utils-modulemap/Utils.modulemap"],
  sdk_includes = [],
  deps = ["//iOSApp/Source/CoreUtilsObjC:CoreUtilsObjC", "//iOSApp/Source/UtilsSwift:UtilsSwift", "//iOSApp/Source/Utils:Utils_public_hmap"] + select({"@build_bazel_rules_ios//:virtualize_frameworks": ["//iOSApp/Source/Utils:Utils_swift_vfs", "//iOSApp/Source/Utils:Utils_vfs"], "//conditions:default": ["//iOSApp/Source/Utils:Utils_swift_vfs", "//iOSApp/Source/Utils:Utils_vfs"]}),
  sdk_frameworks = [],
  weak_sdk_frameworks = [],
  sdk_dylibs = [],
  copts = [] + [] + select({"@build_bazel_rules_ios//:virtualize_frameworks": ["-ivfsoverlay$(execpath :Utils_vfs)", "-F/build_bazel_rules_ios/frameworks"], "//conditions:default": ["-ivfsoverlay$(execpath :Utils_vfs)", "-F/build_bazel_rules_ios/frameworks"]}) + ["-I$(execpath :Utils_public_hmap)", "-fmodules", "-fmodule-name=Source_Utils", "-I."] + select({"@build_bazel_rules_ios//:use_global_index_store": ["-index-store-path", "bazel-out/_global_index_store"], "//conditions:default": ["-index-store-path", "$(GENDIR)/iOSApp/Source/Utils/rules_ios_objc_library_Utils_objc.indexstore"]}),
)
# Rule Utils_objc instantiated at (most recent call last):
#   /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/iOSApp/Source/Utils/BUILD:6:26                                             in <toplevel>
#   /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/external/build_bazel_rules_ios/rules/framework.bzl:97:28 in apple_framework
#   /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/external/build_bazel_rules_ios/rules/library.bzl:1039:24 in apple_library
# Rule objc_library defined at (most recent call last):
#   /virtual_builtins_bzl/common/objc/objc_library.bzl:87:20 in <toplevel>

Reproduction steps
Branch: https://github.com/mattrobmattrob/rules_xcodeproj/tree/mr/rules_ios/lldb.issue.in.tests
Comparison: https://github.com/MobileNativeFoundation/rules_xcodeproj/compare/main...mattrobmattrob:rules_xcodeproj:mr/rules_ios/lldb.issue.in.tests?expand=1

cd examples/rules_ios && bazel run :xcodeproj
open rules_ios.xcodeproj
Select iOSAppMixedUnitTests scheme.
Drop breakpoint in SwiftUnitTests.swift::8 and run SwiftUnitTests.testExample
po self

Expected behavior
Expected:
Test Suite 'Selected tests' started at 2023-05-16 11:26:42.212
Test Suite 'iOSAppMixedUnitTests.xctest' started at 2023-05-16 11:26:42.212
Test Suite 'SwiftUnitTests' started at 2023-05-16 11:26:42.212
Test Case '-[iOSAppMixedUnitTests.SwiftUnitTests testExample]' started.
(lldb) po self
-[SwiftUnitTests testExample]

Demonstrated:
Test Suite 'Selected tests' started at 2023-05-16 11:21:35.141
Test Suite 'iOSAppMixedUnitTests.xctest' started at 2023-05-16 11:21:35.142
Test Suite 'SwiftUnitTests' started at 2023-05-16 11:21:35.142
Test Case '-[iOSAppMixedUnitTests.SwiftUnitTests testExample]' started.
(lldb) po self
error: expression failed to parse:
unknown error
(lldb) po self
warning: Swift error in scratch context: /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/iOSApp/Source/Utils/Foo.h:4:9: note: while building module 'Source_UtilsSwift' imported from /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/iOSApp/Source/Utils/Foo.h:4:
@import Source_UtilsSwift;
        ^

<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/UtilsSwift-umbrella.h"
        ^

warning: /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/bazel-out/ios-sim_arm64-min15.0-applebin_ios-ios_sim_arm64-dbg-ST-d98dbd8343b5/bin/iOSApp/Source/UtilsSwift/UtilsSwift-modulemap/UtilsSwift-umbrella.h:19:1: warning: umbrella header for module 'Source_UtilsSwift' does not include header 'Source_UtilsSwift-Swift.h'

^

<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/Utils-umbrella.h"
        ^

/Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/bazel-out/ios-sim_arm64-min15.0-applebin_ios-ios_sim_arm64-dbg-ST-d98dbd8343b5/bin/iOSApp/Source/Utils/Utils-modulemap/Utils-umbrella.h:16:9: note: in file included from /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/bazel-out/ios-sim_arm64-min15.0-applebin_ios-ios_sim_arm64-dbg-ST-d98dbd8343b5/bin/iOSApp/Source/Utils/Utils-modulemap/Utils-umbrella.h:16:
#import "Foo.h"
        ^

error: /Users/matt.robinson/Developer/github/buildbuddy-io/rules_xcodeproj/examples/rules_ios/bazel-output-base/rules_xcodeproj.noindex/build_output_base/execroot/__main__/iOSApp/Source/Utils/Foo.h:11:31: error: no type or protocol named 'FileManagerProtocol'
- (void)handleFileManager:(id<FileManagerProtocol>)fileManager;
                              ^

error: could not build Objective-C module 'Source_Utils'
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/Utils-umbrella.h"
        ^

.
Shared Swift state for iOSApp_ExecutableName has developed fatal errors and is being discarded.
REPL definitions and persistent names/types will be lost.

error: expression failed to parse:
unknown error

rules_xcodeproj version
5195826c267fb9b5b14e4481a14a3e90231fa0ce
Xcode version
14.3
Bazel version
6.2.0
rules_apple version
No response
rules_swift version
No response
Additional information
No response

The way clang opts are collected, both how Xcode does it, and how we do it manually with swift_debug_settings.py, is by transitively picking up -Xcc flags on swift dependencies. If a dependent swift_library doesn't set this needed -Xcc -ivfsoverlay, then we won't see it, and lldb will be broken. We can't look at the copts of non swift dependencies. So to fix this, a swift_library that depends on Utils_objc would need to set copts =  ["-Xcc" "-ivfsoverlay$(execpath :Utils_vfs)"].

The fact that this compiles though, and only errors in lldb, is concerning. That would indicate to me that we aren't picking up something correctly, because otherwise normal compilation of iOSApp_ExecutableName should fail with the errors you are seeing.

My bet is that it's related to the handling of the apple_framework_packaging (non_xcode_targets) rule in rules_ios that's just the odd exporter of providers compared to the "normal" swift_library => objc_library sandwich (like in experimental_mixed_language_library). But will try to confirm that explicitly tomorrow.

I'm hacking in a fix for this right now until I can get a real solution, it's just running this on the generated {config}-swift_debug_settings.py:
$ sed -i '' \
    's/_swift_vfs.yaml/_vfs.yaml/g' \
    'path/to/xcode/Project.xcodeproj/rules_xcodeproj/bazel/Debug-swift_debug_settings.py'

Just one other data point here. We're hitting a very similar err when stopping at a breakpoint in a test and trying to po, similar err msg mentioning a-umbrella.h file.
Doing the sed cmd above makes po work but auto-complete continues to be broken in the Xcode console, e.g. this shows "No Completions": po foo.<press ESC to show list of options>

Just one other data point here. We're hitting a very similar err when stopping at a breakpoint in a test and trying to po, similar err msg mentioning a-umbrella.h file.
Doing the sed cmd above makes po work but auto-complete continues to be broken in the Xcode console, e.g. this shows "No Completions": po foo.<press ESC to show list of options>

The auto-complete issue is most likely unrelated to this issue, I fixed that by adding the two lines below here:
  "bi $BAZEL_TARGET_ID"
  "bp $BAZEL_TARGET_ID"

Will see if I can find a small repro since it's happened internally file a separate issue to track.

bi shouldn't be needed, it just creates the file that says where to find unit files to import.
bp is interesting. I think it has a side effect of causing some intermediate targets to be compiled/downloaded, which makes me think it's #2023.

bi shouldn't be needed, it just creates the file that says where to find unit files to import.
bp is interesting. I think it has a side effect of causing some intermediate targets to be compiled/downloaded, which makes me think it's #2023.

Yep, confirmed that if I remove bi it continues to work for me, so yeah it's bp that's fixed it. Will take a look at the linked PR. With the sed in this issue and the bi line all our internal test cases that cover the common developer workflows involving LLDB are 🟢 btw 🎉.

Nope, @thiagohmcruz, haven't tried with the incremental generator. Happy to close this if the repro is fixed though!

If it's fixed with the new generation mode, let's leave it open until it's in main (when https://github.com/MobileNativeFoundation/rules_xcodeproj/milestone/42 is closed).


TiffWriter: add an interval, as well as an offset

Description
In the TiffWriter there is a counter_offset which allows you to save images with names 2,3,4,5,... or 11,12,13,14,....:
https://github.com/TomographicImaging/CIL/blob/f8b513b53b44a3f7d7ee7507250965eaffb88d1c/Wrappers/Python/cil/io/TIFF.py#L53-L70
I would like a counter_interval which allows you to save e.g. 2,4,6,8,10... or 1,6,11,16,....
This would be useful for saving outputs of iterative methods every set number of iterations

I think I got round this...


v1.0 backports 2018-06-15

#4527
#4439

This change is

test-me-please

(Ok, looks like previous CI run was a flake. https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/4179/testReport/)


core: Support tokio on local futures

Summary
This PR adds a hack also used by async-std to archive compatibility with tokio. More or less this just means entering the tokio context before running the application.
@edfloreshz can you maybe test this with Done? This should remove the need to wrap tokio-depndant futures into relm4::spawn().
Checklist

[x] cargo fmt
[x] cargo clippy
[x] cargo test
[x] updated CHANGES.md

Sure! Feel free to request me a review whenever 👍🏼 I'll let you know what I find


CIV-0000 create latest update builder

Before creating a pull request make sure that:

[x] commit messages are meaningful and follow good commit message guidelines
[x] README and other documentation has been updated / added (if needed)
[x] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link
Change description
Create class latestUpdateBuild to make more easy the creating lastUpdate screens
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

are there any tests needed for these changes? either for the view of the screen depending on the if's or for the js methods added?

are there any tests needed for these changes? either for the view of the screen depending on the if's or for the js methods added?
Yes, I added the unit test.


windows 集成终端打开路径bug

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:  1.84.2
OS Version:  windows11

Steps to Reproduce:
这个bug在终端未打开时不会发生，但是当我已经启动了其他终端后

当我在内置终端打开时，路径是正确的

但当我windows终端打开，目录指向不对 /c/windows/System32

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.85.1. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

感谢您创建此问题！
然而，由于您没有使用英语，我们很难进行处理。如果可能，请您将此问题描述修改为英文。请您直接编辑此问题，不要提交回复或新问题。
请注意，机器翻译服务通常无法处理用于讨论软件问题所需的技术性语言，所以我们推荐您寻找一位熟悉相关技术的人来协助翻译。
如果您无法做到，没有关系。此问题已经被标记为需要翻译，等待社区成员志愿进行翻译。

Thanks for creating this issue!
Unfortunately, as the description is not in English, it's hard for us to work on. If possible, please edit the original issue to be in English.
In our experience, automatic translation services are not able to handle the technical language needed to effectively communicate about software issues, so we recommend finding a human translator who is familiar with our technology to help.
Otherwise, we have marked this as needing translation and a community member may be able to help out.


Update README.md

Add reference to https://github.com/Acurast/data-processor repo to README.

@godenzim was my addition incorrect?


fix(dis7): switch the size of length and status

It looks like the total size of the fields were maintained but the wrong sizes to each one,
the PduHeader is technically not supposed to be used as the Pdu class can be used directly which inherits from the PduSuperClass.

Thank-you @kurtsansom yes I believe you're right.
This is what I see in the spec.

 And like you said this change also matches the data type in the pduSuperClass so all the more reason.
Nice catch!

Thank-you @kurtsansom yes I believe you're right.
This is what I see in the spec.

 And like you said this change also matches the data type in the pduSuperClass so all the more reason.
Nice catch!


enable Eslint 2.5 cache

Once it will be stable.

It will never work with gulp-eslint I suppose, never mind.


Instructions for installing from source?

I'd like to try and get this working and contribute myself. Are there instructions for installing this from github rather than the clion plugins window?

Installation instruction should be very simple. You can either use Intellij Idea Ultimate if you have a license or use intellij community edition which you can get for free here https://www.jetbrains.com/idea/download
When installing make sure to have VCS plugins enabled and then you should be able to import the repository from git. To compile you'll need to add CLion Platform Plugin SDK. Follow this guide here and just specify the CLion path instead of the Intellij one they use https://www.jetbrains.org/intellij/sdk/docs/basics/getting_started/setting_up_environment.html. To run and debug you'll want to create a run and debug configuration for CLion which you can find more about here. https://www.jetbrains.com/help/idea/run-debug-configuration-plugin.html. Here you need to set CLion as JRE. This will start a CLion instance in a sandbox enviroment.
Think that should be all


Nick: Watch<PHONE_NUMBER>2

Nick requests the watch of the watch_number<PHONE_NUMBER>2. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Approved by Spevacus in Charcoal HQ


kwin_wayland: High GPU usage when moving pointer

Describe the bug
High GPU usage when moving pointer, only on NixOS.
Steps To Reproduce
Steps to reproduce the behavior:
0. Maybe testing only on Intel HD Graphics ≈5500 will show similar results.

Enable plasma6
Boot to wayland session
Install and launch intel_gpu_top
Move pointer fast
See high gpu usage in render/3d

Expected behavior
GPU usage should definitely be lower.
Screenshots
Nixos - plasma 6:
https://github.com/user-attachments/assets/e06dec89-b257-4a02-9a7e-c6a01f75eb09
Arch:
https://github.com/user-attachments/assets/bcaaf7da-27a6-4ed5-b184-3e39df528f0f
Nixos - Plasma5:
GPU usage is almost the same as in Arch, but when the pointer moves, systemd uses the gpu.
Additional context
Changing kernels does nothing, power-profiles-deamon installed on both systems and running from battery.
On x11 GPU usage is normal.
CPU is overheating while browsing google 😃
I have PTM7950 mounted, in x11 cpu is cold (max 55*C while heavy usage)
Notify maintainers
@K900

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.39, NixOS, 24.11 (Vicuna), 24.11.20240714.693bc46`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/store/iqxdbcjlg9wx8gdc8bdhy9nsd5imcbjp-source`

Add a :+1: reaction to issues you find important.

I checked how GPU usage is with latest unstable iso and usage is perfect, so issue is in my setup.
I didn't find what option in my config caused that high GPU usage, so I decided to reinstall nixos freshly and looks like that fixes problem. Maybe some old files from plasma5 caused this.


Add plugin copy/paste to manage OS clipboard input/output

The idea is to propose a plugin to manage interactions with the operating system clipboard.
This feature have a high value for UI designer, but in Pharo the way to manage the clipboard is not "clear".
Actually for Windows we have a project to do that (https://github.com/OpenSmock/OS-Windows), and it works nicely.
But I don't know how to do that for others platforms.
May be we need to have a common API, OS agnostic, to manage that.
In consequence this is necessary to propose a Pyramid plugin with a good level of front-end API to manage that, because the clipboard management backend can change depending decision of that in Pharo. This clipboard management can be implemented in an other project to support the interaction with the OS. The Pyramid plugin can use this project to exploit the clipboard. This is a reflexion to have with @ELePors.
About features in Pyramid, copying a BlElement should fill the clipboard with theses capacities:

A bitmap image of the BlElement (PNG, JPEG, etc.) => to paste this element in Word, Excel, Mail, Figma, etc.
A text representation of the BlElement (i.e. some properties) => to paste this element in a notepad
A SVG representation of the BlElement => to paste this element in a design tool like Figma, Inkscape, Adobe Illustrator, etc.
A bit Pyramid version of the BlElement => to paste this element in another Pyramid projet => to paste this element in Word, Excel, Mail, etc.
[Futur] - An HTML version of the BlElement (with the bitmap image)

Actual use-cases:
Copy a BlElement from Pyramid and paste it in a mail as an image.
@Nyan11 what is actually available ?

You can add custom menu in a plugin with:
"Create a Pyramid plugin"

Object << #PyramidQuickMenu
	traits: {TPyramidPlugin};
	slots: {};
	tag: 'plugin-space';
	package: 'Pyramid-Bloc'

"In the connectOn: method do ->"
connectOn: aPyramidEditor

	(aPyramidEditor window services at: #selectionMenu) addItem: [
		:aBuilder | self menuFor: aBuilder ]

"In the menuFor: do ->"

menuFor: aBuilder

	aBuilder
		addGroupEmptySelection: [ :group :empty |
			group
				addItem: [ :item |
					item
						name: 'i am a menu for nothing';
						enabled: true;
						action: [ self inform: 'nothing' ];
						yourself ];
				yourself ]
		order: 40.
	aBuilder
		addGroupSingleSelection: [ :group :single |
			group
				addItem: [ :item |
					item
						name: 'i am a menu for single';
						enabled: true;
						action: [ self inform: single printString ];
						yourself ];
				yourself ]
		order: 40.
	aBuilder
		addGroupMultiSelection: [ :group :multi |
			group
				addItem: [ :item |
					item
						name: 'i am a menu for mutli';
						enabled: true;
						action: [ self inform: multi size printString ];
						yourself ];
				yourself ]
		order: 40

You can also export a Bloc design as a Form and then transform the Form as a png. I do not know if others export format exist.


To allow disambiguation, bind a global identifier in global namespace that refers to itself

Implementation issue for #781
This should work:
namespace A {
  namespace B {
    model Y extends global.B.X {}
  }
}

namespace B {
  model X {}
}

Playground Link

Priority: 2
Estimate: 5


db/Command getRawSql boolean values doesn't work

Hi. There is no boolean check in getRawSql function (db/Command line 155), so 'false' values converts to empty string, which cause an syntax error (at least in pgsql), e.g.:
UPDATE "city" SET "deletion_mark"=,

Fix should look like (line 163):
 if (is_string($value)) {
        $params[$name] = $this->db->quoteValue($value);
} elseif (is_bool($value)) {                            // <= fix
        $params[$name] = ($value ? 'TRUE' : 'FALSE');   // <= fix
} elseif ($value === null) {
        $params[$name] = 'NULL';
} elseif (!is_object($value) && !is_resource($value)) {
        $params[$name] = $value;
}

Fixed. Thanks for reporting.
btw., here's the quote from that method docs:

Note that the return value of this method should mainly be used for logging purpose.
It is likely that this method returns an invalid SQL due to improper replacement of parameter placeholders.

So don't rely on SQL produced. Use $command->sql and bound parameters instead.


Python wrapper performance issue

Hello,
I start with a ResNet50 onnx file or a VGG19 ONNX file. I compile it to generate a SO file. Everything is working fine.
Now I want perform two things : 1) load/initialize the model in memory and 2) Fast predict with it when data samples are incoming.
I implement it copy/pasting the doc but I am hurting critical performance issue. The initiIialization step is very fast (too much <0.01sec), and the prediction is very slow compared to other inference framework (several minutes). I suspect ExecutionSession constructor to not loading and initializing the model, and the run method to load and run it instead of only predict.
How can I load/initialize once and perform fast predictions each time new data are incoming ?
import numpy as np
from PyRuntime import ExecutionSession
import time
path='../models_lib/ONNX/VGG19_O3.so'
batch_data=np.random.uniform(0,1,(32,224,224,3)).astype(np.float32)

st=time.time()
session=ExecutionSession(path, "run_main_graph")
print("load and initialization time: ",time.time()-st) # < 0.01 seconds

st=time.time()
y=session.run([batch_data])
print("prediction time: ",time.time()-st) # Take 678 seconds !!!!

Your observations are correct, we do not have to do anything during loading/initialization because all of the constants, for example, are loaded directly from the binary.
We currently don't optimize conv ops, which is the predominant performance component of ResNet and other conv based networks. We are working on it. If you are interested in contributing to the performance of ONNX-MLIR, we welcome the contributions from all.

Hi @AlexandreEichenberger, Thanks for sharing those details.
I was recently exploring this project and tested the inference of the generated model. I tested efficientnet_b0 from torchvision.
On my machine which is old Intel i3 5th Gen,

Pytorch model takes 122 ms ± 32.3 ms per loop (mean ± std. dev. of 20 runs, 10 loops each)
Onnx model takes 64.2 ms ± 10.5 ms per loop (mean ± std. dev. of 20 runs, 10 loops each)
Onnx-MLIR model takes 1.2 s ± 55 ms per loop (mean ± std. dev. of 20 runs, 10 loops each)

I'm wondering if I'm doing something wrong or this is the expected inference speed on the mentioned model?
On what kind of models is this project supposed to be used to get the faster inference?
Thanks


GP ungracefully fails handling a single choice in search space

When using GP, if you have an enum with a single choice, the GP strategy fails with the following message:
Traceback (most recent call last):
  File "/Users/zzz/anaconda/envs/osprey/bin/osprey", line 9, in <module>
    load_entry_point('osprey', 'console_scripts', 'osprey')()
  File "/Users/zzz/Projects/github/osprey/osprey/cli/main.py", line 37, in main
    args_func(args, p)
  File "/Users/zzz/Projects/github/osprey/osprey/cli/main.py", line 42, in args_func
    args.func(args, p)
  File "/Users/zzz/Projects/github/osprey/osprey/cli/parser_worker.py", line 8, in func
    execute(args, parser)
  File "/Users/zzz/Projects/github/osprey/osprey/execute_worker.py", line 77, in execute
    sessionbuilder=config.trialscontext)
  File "/Users/zzz/Projects/github/osprey/osprey/execute_worker.py", line 102, in initialize_trial
    params = strategy.suggest(history, searchspace)
  File "/Users/zzz/Projects/github/osprey/osprey/strategies.py", line 276, in suggest
    X, Y, V, ignore = self._get_data(history, searchspace)
  File "/Users/zzz/Projects/github/osprey/osprey/strategies.py", line 232, in _get_data
    point = searchspace.point_to_gp(param_dict)
  File "/Users/zzz/Projects/github/osprey/osprey/search_space.py", line 112, in point_to_gp
    return [var.point_to_gp(point_dict[var.name]) for var in self]
  File "/Users/zzz/Projects/github/osprey/osprey/search_space.py", line 234, in point_to_gp
    return float(index) / (len(self.choices) - 1)
ZeroDivisionError: float division by zero

From the config:
search_space:
  # the search space is specified by listing the variables you want to
  # optimize over and their bounds for float and int typed variables,
  # or the possible choices for enumeration-typed variables.
  featurizer__types:
    choices:
      - ['phi', 'psi']
      #- ['phi', 'psi', 'chi1']
    type: enum

Thanks for pointing this out- the error message should be more helpful.
As a workaround, if you are restricting any part of your model to a single choice, you can initialize it that way at the beginning of your pipeline and remove it from your search space, e.g. for the types of the dihedral featurizer,
estimator:
    eval: |
        Pipeline([
                ('featurizer', DihedralFeaturizer(types=['phi', 'psi'], sincos=True)),
                ('cluster', MiniBatchKMeans()),
                ('msm', MarkovStateModel(n_timescales=5, verbose=False)),
                                    ])

As a little bit of a hack when I have wanted to create a model that isn't actually searching anything I would do something like:
search_space:
  featurizer__types:
    choices:
      - ['phi', 'psi']
      - ['phi', 'psi']
    type: enum

fixed in #191


The "Open a Sample Project" links don't work

The "Open a Sample Project" links produce a "Create Project" dialog instead of opening the installed samples.

I don't see this. Is this a build you've made yourself, or the official release?

The official release.

What should happen (and what I get) is that you get a save dialog to say where the sample project should be copied to, and after choosing a location, the sample project opens in Inform.

OK. It works as you described in both the official release and the compiled from source versions. It was just a bit counterintuitive to see a create project dialog in response to "Open a Sample Project". Maybe the installer/app could copy the samples to the working directory structure on installation/first use? Just a suggestion.

I have re-worded the text on that save dialog to hopefully make things clearer. I'm reluctant to change the behaviour beyond that as this is how the macOS front-end works, which I tend to try to follow with the Windows front-end, where practical.


feat(indexRootPaths): invalidate index.html root paths

Haven't tested this yet, will tomorrow during the work day. Works, see below 🐱
Fixes #9

@lpender This PR seems to be working beautifully for us.  Here is a my test case:
Gulp Task
Note indexRootPath: true
gulp.task('publish:aws', (cb) => {
  const publishHeaders = {
    'Cache-Control': 'max-age=315360000, no-transform, public',
    'x-amz-acl': 'public-read',
  }

  const s3Opts = {
    accessKeyId,
    secretAccessKey,
    params: {
      Bucket: paths.publish.s3Bucket,
    },
  }

  const cloudfrontOpts = {
    accessKeyId,
    secretAccessKey,
    distribution,
    indexRootPath: true,       // <-- Here's the new option
  }

  const publisher = g.awspublish.create(s3Opts, {
    cacheFileName: '.awspublish-cache'
  })

  return gulp.src(paths.publish.s3Src)
    .pipe(awspublish.gzip())
    .pipe(parallelize(publisher.publish(publishHeaders), 10))
    .pipe(publisher.sync())
    .pipe(cloudfrontInvalidateAwsPublish(cloudfrontOpts))
    .pipe(publisher.cache())
    .pipe(awspublish.reporter())
})

CloudFront Invalidation
Notice all the index.html root paths were invalidated.  The .xml files can be seen as not invalidating their root path /.
/about-us/
/about-us/index.html
/careers/
/careers/index.html
/contact-us/
/contact-us/index.html
/feed-50d794812f.xml
/feed-bc5f5fe623.xml
/press/
/press/index.html
/press/example-post/
/press/example-post/index.html
/press/releases/
/press/releases/index.html
/press/releases/example-press-release/
/press/releases/example-press-release/index.html
/press/releases/hot-off-the-press/
/press/releases/hot-off-the-press/index.html
/press/resources/
/press/resources/index.html
...

Bump @lpender

Thanks for the PR

And the careful testing/documenting

Published 0.2.0


Failed to load assistants

For some reasons l`m getting this error, have double checked export TAVILY_API_KEY
Please verify if the API server is running or accessible from the browser.
TypeError: Failed to fetch

Did your server successfully start?

issue fixed  Disable Ad Blockers or Privacy Extensions
Temporarily disable any ad-blocking or privacy-protection extensions in your browser.
Example: For uBlock Origin, click the extension icon and disable blocking for the site.


Cooldown for certain commands

implement my own conditional cooldown instead of @commands.cooldown()

Reverted to using @commands.cooldown()


Enhanced type safety with validations.

Removed assertions that are now redundant with type constraints.
Type safety will trigger a compile time error rather than a runtime error.

Codecov Report

Merging #541 into version_4 will decrease coverage by 0.54%.
The diff coverage is 76.19%.

@@              Coverage Diff              @@
##           version_4     #541      +/-   ##
=============================================
- Coverage      30.80%   30.26%   -0.55%
=============================================
  Files             37       37
  Lines           1558     1543      -15
=============================================
- Hits             480      467      -13
+ Misses          1078     1076       -2

Impacted Files
Coverage Δ

lib/src/form_builder_field.dart
80.70% <ø> (ø)

lib/src/form_builder_validators.dart
84.50% <76.19%> (+0.78%)
:arrow_up:

lib/localization/form_builder_localizations.dart
75.60% <0.00%> (-2.44%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbd8c54...d22645d. Read the comment docs.


"error: Could not find module 'Data.Aeson.Key'" preventing compile/build

When I attempt stack build in the root directory of the project, I get the following error:
servant-server-base> /home/jayjader/src/sensei/servant-server-base/src/Data/Aeson/Extra.hs:14:1: error:
servant-server-base>     Could not find module ‘Data.Aeson.Key’
servant-server-base>     Perhaps you meant
servant-server-base>       Data.Aeson.TH (from aeson-<IP_ADDRESS>)
servant-server-base>       Data.Aeson.Text (from aeson-<IP_ADDRESS>)
servant-server-base>       Data.Aeson (from aeson-<IP_ADDRESS>)
servant-server-base>     Locations searched:
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/Data/Aeson/Key.hs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/Data/Aeson/Key.lhs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/Data/Aeson/Key.hsig
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/Data/Aeson/Key.lhsig
servant-server-base>       src/Data/Aeson/Key.hs
servant-server-base>       src/Data/Aeson/Key.lhs
servant-server-base>       src/Data/Aeson/Key.hsig
servant-server-base>       src/Data/Aeson/Key.lhsig
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/autogen/Data/Aeson/Key.hs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/autogen/Data/Aeson/Key.lhs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/autogen/Data/Aeson/Key.hsig
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/autogen/Data/Aeson/Key.lhsig
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/global-autogen/Data/Aeson/Key.hs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/global-autogen/Data/Aeson/Key.lhs
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/global-autogen/Data/Aeson/Key.hsig
servant-server-base>       .stack-work/dist/x86_64-linux-tinfo6/Cabal-<IP_ADDRESS>/build/global-autogen/Data/Aeson/Key.lhsig
servant-server-base>    |
servant-server-base> 14 | import Data.Aeson.Key (fromText)
servant-server-base>    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm on arch-linux, using the stack-static Arch User Repository package.
When googling the error, I noticed that the most recent version of aeson seems to be 2.2. I don't know haskell very well, but I couldn't find any version pinning of aeson by grep-ing in the project files. So I'm not quite sure why the build tool mentions aeson-<IP_ADDRESS> in the error message...
The full log of the stack build invocation is here in a gist.

First of all, thanks a lot for trying sensei!
I am not using stack on a regular basis anymore so it might have bitrotten. I have upgraded to a more recent stockage version on master, could you give it a try?

First of all, sorry for the long delay.
I tried rebuilding today with an up-to-date clone of this repo. Aside from needing to npm install inside ui/ (which complains about missing peer dependencies, by the way), stack build at the project root worked perfectly.


As a developer, I want to ensure there are no TODO's left when handing in the project, in order to avoid flunking the exam

Acceptance criteria:
There are no TODO's in the main branch

Also ensure there are no unused methods in the repositories or cheepservice

And check for stupid-aaa comments


rebasing ocs-ci and correcting ocs-ci-03 patch

Rebasing ocs-ci and correcting typo errors in one of the patches.

Thanks


"Advanced X" Feats Missing Choice Set

The Advanced series of archetype feats (e.g., Advanced Trickery, Advanced Fury, Advanced Shooting, etc.) seem to be missing their Choice Set and Grant Item rules.  They should work similarly to the Basic series (e.g., Basic Trickery, Basic Fury) but with a feat equivalent of half the character's level.  An example of the feat text is here: https://2e.aonprd.com/Feats.aspx?ID=674

Duplicate of #1956, closing


[v1.17.0]  Automerge / Admin merge sometimes doesn't work

Description
I've two issues^1 which are closely related with latest gitea v1.17.0

I can't see the automerge feature, but normally i can force merge as admin.

Today i couldn't merge PR's because they where blocked because they are outdated

Gitea Version
1.17.0
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots

Git Version
offical docker image
Operating System
offical docker image
How are you running Gitea?
2 node HA from official gitea chart with redis / nfs
Database
PostgreSQL

After updating the protection rule i could merge ad admin again.

I also saw PR's stuck in merge conflict check

sometimes i also see broken fork information after using the merge update button

Ok, i need to enable at least two merge methods to get the dropdown for automerge

I think this is the same as #24445 and should be fixed by #30780


Init atomic-server module

Building on work done in https://github.com/ngi-nix/ngipkgs/pull/170 (but not requiring it), this PR inits a module for atomic-server.
It's not ready for review but tested locally.

Alright! After many many months, I think this may finally be ready for review.
We generate the .env file from settings, the systemd unit works on start up, and there is a test that passes.
The main thing to note, is that the .env file does end up in the /nix/store, though there aren't any secrets in the .env file, so I think that's okay? Alternatively we could get the user to provide the environmentFile as a secret that doesn't end up in the nix store.

I'm not sure why the overview CI check is failing, or even what it does and how it's related to this PR.
I will run a nixfmt once I'm back at my computer to fix the pre-commit checks.


fix warning reported by linter

Description
fix warning reported by linter - unused function.

/test pre-serverless-controller-lint


Add instructions for installing xFormers on linux

I've written up the install procedure for xFormers on Linux systems.
I need help with the Windows install; I don't know what the build dependencies (compiler, etc) are. This section of the docs is currently empty.
Please see docs/installation/070_INSTALL_XFORMERS.md

xFormers has precompiled wheels available for Windows/Linux that work with PyTorch 1.13.1 which can be installed with this command: pip install xformers==0.0.16rc425
Wouldn't it be easier for users to simply install this package as opposed to building it themselves?
I've tested the version I've mentioned above and it works fine with diffusers 0.11.1 and with this repo.

xFormers has precompiled wheels available for Windows/Linux that work with PyTorch 1.13.1 which can be installed with this command: pip install xformers==0.0.16rc425
Wouldn't it be easier for users to simply install this package as opposed to building it themselves?
I've tested the version I've mentioned above and it works fine with diffusers 0.11.1 and with this repo.

This certainly would - thanks for sharing! I’ll give it a test on 2.3 tonight and see if it “just works” ™

xFormers has precompiled wheels available for Windows/Linux that work with PyTorch 1.13.1 which can be installed with this command: pip install xformers==0.0.16rc425
Wouldn't it be easier for users to simply install this package as opposed to building it themselves?
I've tested the version I've mentioned above and it works fine with diffusers 0.11.1 and with this repo.

I can confirm on ubuntu 22.04 with main branch installing
"pip install xformers==0.0.16rc425"
"pip install triton"
makes SD 2.1 float16 work
python -m xformers.inf output
memory_efficient_attention.cutlassB:               available
memory_efficient_attention.flshattF:               available
memory_efficient_attention.flshattB:               available
memory_efficient_attention.smallkF:                available
memory_efficient_attention.smallkB:                available
memory_efficient_attention.tritonflashattF:        available
memory_efficient_attention.tritonflashattB:        available
swiglu.fused.p.cpp:                                available
is_triton_available:                               True
is_functorch_available:                            False
pytorch.version:                                   1.13.1+cu116
pytorch.cuda:                                      available
gpu.compute_capability:                            8.6
gpu.name:                                          NVIDIA GeForce RTX 3060
build.info:                                        available
build.cuda_version:                                1107
build.python_version:                              3.10.9
build.torch_version:                               1.13.1+cu117
build.env.TORCH_CUDA_ARCH_LIST:                    5.0+PTX 6.0 6.1 7.0 7.5 8.0 8.6
build.env.XFORMERS_BUILD_TYPE:                     Release
build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS:        None
build.env.NVCC_FLAGS:                              None
build.env.XFORMERS_PACKAGE_FROM:                   wheel-v0.0.16rc425
source.privacy:                                    open source```

I'm light of the very welcome news that you can easily pip install on Linux, I'm going to rewrite the doc a bit. So there's no wheel install for Windows?

There is a Windows wheel, I'm using it at this very moment. It is also listed here: https://pypi.org/project/xformers/0.0.16rc425/#files

Do you have any tips or tricks for using the Windows wheel or was it a
smooth experience?
On Fri, Jan 20, 2023 at 10:34 AM SnakePin @.***> wrote:

There is a Windows wheel, I'm using it at this very moment. It is also
listed here: https://pypi.org/project/xformers/0.0.16rc425/#files
—
Reply to this email directly, view it on GitHub
https://github.com/invoke-ai/InvokeAI/pull/2360#issuecomment-1398568441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA3EVLFQEO25UYELUD7BQTWTKWB3ANCNFSM6AAAAAAT7FBAWI
.
You are receiving this because you authored the thread.Message ID:
@.***>

Do you have any tips or tricks for using the Windows wheel or was it a smooth experience?

The only requirement is that the VC++ 2015-2022 redistributable must be installed, as it's compiled with Visual C++ 2019
This shouldn't be a problem as torch requires the same redist too.

pip install xformers==0.0.16rc425 pip install triton
python -m xformers.inf output

I've added the pip install instructions. Thanks very much for this! I wish I'd known that there were pip wheels available, cause that would have saved me a lot of hair-tearing a few days ago.

I just wanted to comment that pinning xformers==0.0.16rc425 is not a viable solution because rc versions WILL be deleted from pypi as when the storage starts to get full.
(which is often if every wheel is 50MB and you build 20 different version).
Installing the latest available rc version can be done by
pip install -U --pre xformers

This will make sure the installation does not break, as long as 0.0.16 is not officially released yet, then we can pin to 0.0.16.


users/set_password/ options seems wrong

This is the response i get when making an options request to that endpoint. the actual expected arguments seem to be current password and new password, however this is what i get from options which is weird.
Currently running djoser version 2.0.3

This seems to be the case for all endpoints. This, afaik, shouldn't show the returned serializer but rather the endpoints..
This is probably because DRF bases it off of serializer_class, but that shouldn't be the case for your codebase

This is happening, because of the following:
    def get_serializer_class(self):
        if self.action == "create":
            if settings.USER_CREATE_PASSWORD_RETYPE:
                return settings.SERIALIZERS.user_create_password_retype
            return settings.SERIALIZERS.user_create
        elif self.action == "destroy" or (
            self.action == "me" and self.request and self.request.method == "DELETE"
        ):
            return settings.SERIALIZERS.user_delete

        ...

        elif self.action == "me":
            return settings.SERIALIZERS.current_user

        return self.serializer_class

DRF view is initialized with action equal to metadata:
    def initialize_request(self, request, *args, **kwargs):
        """
        Set the `.action` attribute on the view, depending on the request method.
        """
        request = super().initialize_request(request, *args, **kwargs)
        method = request.method.lower()
        if method == 'options':
            # This is a special case as we always provide handling for the
            # options method in the base `View` class.
            # Unlike the other explicitly defined actions, 'metadata' is implicit.
            self.action = 'metadata'
        else:
            self.action = self.action_map.get(method)
        return request

that's why self.serializer_class is always taken. One of the solutions probably would be to make the serializer class selection based on the url_path and request method. Any suggestions appreciated, if we come up with a good approach, it shouldn't take a lot of time to address this issue.

So i would argue that the issue should be addressed at the DRF level. This problem will arise in a multitude of projects, and to the best of knowledge it is the "right" way of doing it (actions, that is). Not sure exactly how they should fix it, but it seems rather important in my opinion. However a hacky solution could be provided in the meantime, but i would advice against it, based on my (though limited) experience.


BUG: breaks kedro-viz (not handling save_version = None)

Describe the bug
When using other kedro commands, like kedro viz, the package brakes.
This is because no save_version is initialized by other commands and the plugin currently doesn't handle the case.
This problem is also existent in the current version. (Traceback is for 0.0.7)
See: https://github.com/neptune-ai/kedro-neptune/blob/f975c03e739db8e8eb5ab45555430ff23b8facc9/kedro_neptune/init.py#L417
Reproduction
Install kedro viz
pip install kedro-viz
Add an exception in the src/my_package_name/settings.py
# List the installed plugins for which to disable auto-registry
DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",)

run
kedro viz

Expected behavior
Kedro viz works as excpeted.
Optimal, It is recognized that this is not a kedro run so no neptune session is initalized.
Minimal,It is recognized that no save_version is present, (catalog is created separately) no run_id is calculated.
Traceback
Command: kedro viz
Traceback:
2022-05-23 16:32:15,384 - kedro.framework.hooks.manager - INFO - Registered hooks from 1 installed plugin(s): kedro-neptune-0.0.7
2022-05-23 16:32:17,411 - kedro_viz.integrations.pypi - INFO - Checking for update...
2022-05-23 16:32:18,141 - kedro.framework.session.store - INFO - `read()` not implemented for `BaseSessionStore`. Assuming empty store.
2022-05-23 16:32:18,340 - ProjectHooks - INFO - register_catalog(self.env='local', save_version=None, load_versions=None, journal=None)
Traceback (most recent call last):
  File "C:\PATH\TO\ENV\lib\site-packages\kedro_viz\launchers\cli.py", line 122, in viz
    run_server(**run_server_kwargs)
  File "C:\PATH\TO\ENV\lib\site-packages\kedro_viz\server.py", line 75, in run_server
    catalog, pipelines, session_store_location = kedro_data_loader.load_data(
  File "C:\PATH\TO\ENV\lib\site-packages\kedro_viz\integrations\kedro\data_loader.py", line 68, in load_data
    catalog = context.catalog
  File "C:\PATH\TO\ENV\lib\site-packages\kedro\framework\context\context.py", line 320, in catalog
    return self._get_catalog()
  File "C:\PATH\TO\ENV\lib\site-packages\kedro\framework\context\context.py", line 383, in _get_catalog
    hook_manager.hook.after_catalog_created(  # pylint: disable=no-member
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\callers.py", line 208, in _multicall
    return outcome.get_result()
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\PATH\TO\ENV\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "C:\PATH\TO\ENV\lib\site-packages\kedro_neptune\__init__.py", line 415, in after_catalog_created
    self._run_id = hashlib.md5(run_id.encode()).hexdigest()
AttributeError: 'NoneType' object has no attribute 'encode'
kedro.framework.cli.utils.KedroCliError: 'NoneType' object has no attribute 'encode'
Run with --verbose to see the full exception
Error: 'NoneType' object has no attribute 'encode'

Additional context
Even when the None case is handled, a new error could arise due to creating a neptune experiment for every run of a kedro viz command.

Hi @king-michael,
Prince Canuma here, a Data Scientist at Neptune.ai,
Thank you for reporting the issue, I have reached out to the engineering team to investigate this. As soon as I have any news or updates from the team, I will let you know!

Hi @king-michael,
Good news!!!
I have talked to the engineering team and we have a new release of the kedro neptune integration (v0.1.1) that allows you to use environment variables.
You can use the environment variables to set Neptune to debug mode or disable it and when you are using kedro viz which I think will solve your problem,
Let me know if this works for you!
Release link:
https://github.com/neptune-ai/kedro-neptune/releases/tag/0.1.1

Hi @king-michael,
I'm just checking in to see if you still need help with this or if you have any other questions.

Hi @Blaizzy,
I'm running into the same problem, with kedro==0.18.2 and kedro-neptune==0.1.1. Can you maybe clarify what the suggested solution would be? Just using enabled: false in neptune.yml doesn't solve the problem for me and still produces the error above. Kedro-viz only works if you temporarily disable hooks for kedro-neptune by adding DISABLE_HOOKS_FOR_PLUGINS =("kedro-neptune") in settings.py

Hey @fabz1,
In the newest release, you can enable and disable Neptune by setting the NEPTUNE_ENABLED environment variable passing either True or False.
Example:
export NEPTUNE_ENABLED='False'
PR: https://github.com/neptune-ai/kedro-neptune/pull/35
Let me know if this works for you

Hi @Blaizzy,
thanks but that doesn't work for me. The only option is to temporarily disable hooks in settings.py while visualizing the pipeline.

No problem,
I will talk to the devs and get back to you,
Meanwhile is there anything else I can help you with?

Hey @king-michael and @fabz1
The fix is on production and your issue should disappear now! 🎉
Let me know if it works for you or not,
Looking forward to hearing from you.


RFC 7232: Directory Service responds with 200 OK when If-Modified-Since

Issue was caused by a difference of less than one second between the value
of the header  If-Modified-Since and the actual date of the modification
of the file.
Additionally the header Last-Modified was not sent when sending the status
304.
I have also took the liberty of closing certain FileInputStream which were
opened but never closed.

@DoruM @sbadugu Please assign this to someone else for review. I am snowed under and will not be available next week since I will be working with a customer.

Thanks Dorum, looks good to me and thanks for fixing the resource close.
Please try to be a little bit tidier with the formatting.  See this repo for formatters.  On nuklei code based these are strictly enforced.

Done.


Returning inherited models of various types

Hello!
I ran into a problem while writing tests for our application. We have a certain backend route that can return various models inherited from a single base model. When I tried to mock it, I found out that the mock replaces the types of the supplied model instances with the type specified as the modelName for the mock, so the data aren't loaded properly.
To illustrate this, I wrote a simple test based on your test setup:
test("mock returns inherited models with proper types", function(assert) {
  const smallHat = make('small-hat');
  const bigHat = make('big-hat');
  let mock = mockQuery('hat', { size: 'all' }).returns({ models: [smallHat, bigHat] });
  let responseData = mock.responseJson.data;
  equal(responseData[0].type, 'small-hat');
  equal(responseData[1].type, 'big-hat');
});

As it turned out, the data type specified in the JSON is 'hat' in both cases.
I would greatly appreciate your suggestions on handling this situation.

I will put this test to use in finding out what is going on ... on first glance it is probably due to the way polymorphism is handled in factory guy and that I have tested in a relationship setting and not freestyle as this one is ( load some polymorphic models on their own )

this issue is now fixed in version 2.7.4 .. thanks for bringing it up

Thank you very much for fixing it! It will help us a lot.

Your 100% welcome. I always like that you give me very clear idea of what is going on and write a test I can work with to fix the issue.

Hello! It seems that this issue strikes again. The polymorphic models are returned just fine when we use buildList (I see that a modified version of the test from my original post is included into the test suite), but the modelName is replaced with the one from the mock definition if we try to use returns({models: ...}). Here's the test I tried to include into the shared-adapter-behaviour.js:
test("using returns with model instances for a polymorphic route", function(assert) {
  run(() => {
    let done = assert.async();
    let bigHat = make('big-hat');

    console.log(bigHat.constructor.modelName); // "big-hat"

    mockQuery('hat', {size: 'all'}).returns({models: [bigHat]});

    FactoryGuy.store.query('hat', {size: 'all'}).then(function(hats) {
      assert.equal(hats.get('length'), 1);

      console.log(hats.get('firstObject').constructor.modelName); // "hat"

      assert.equal(hats.get('firstObject').constructor.modelName, bigHat.constructor.modelName) // fails
      assert.equal(FactoryGuy.store.peekAll('hat').get('content').length, 0, "does not make another hat"); // fails again
      done();
    });
  });
});

It seems that the previous behaviour was broken by this fix. Our test cases started to work just fine when we returned type: model.constructor.modelName into the model definition, but I see that it may break some other use cases. I don't know what to suggest... maybe we just need some kind of config option that would define the preferred behaviour for a mock.

Interesting, I am going to try this out and see if i can figure this out.


Configuration file configs/musdb_config.json not found

Description
When tried to train it by musdb18, it saying the .json file is not found when it is clearly in that folder
Step to reproduce

Installed using Anaconda
Run as:  spleeter train -p configs/musdb_config.json -d /musdb

OSError: Configuration file configs/musdb_config.json not found
Output
Traceback (most recent call last):
File "c:\users\osoho\anaconda3\envs\spleeter-cpu\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "c:\users\osoho\anaconda3\envs\spleeter-cpu\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\osoho\Anaconda3\envs\spleeter-cpu\Scripts\spleeter.exe_main.py", line 7, in
File "c:\users\osoho\anaconda3\envs\spleeter-cpu\lib\site-packages\spleeter_main.py", line 47, in entrypoint
main(sys.argv)
File "c:\users\osoho\anaconda3\envs\spleeter-cpu\lib\site-packages\spleeter_main_.py", line 40, in main
params = load_configuration(arguments.params_filename)
File "c:\users\osoho\anaconda3\envs\spleeter-cpu\lib\site-packages\spleeter\utils\configuration.py", line 45, in load_configuration
raise IOError(f'Configuration file {descriptor} not found')
OSError: Configuration file configs/musdb_config.json not found
Environment
| Windows / Linux / MacOS / other |
| Installation type | Conda /|
| Hardware spec     | i5-7400 ; 8 GB ; NVIDIA GTX1050TI |           |
Additional context

You are using a UNIX filesystem path. You are using Windows. So it would be configs\musdb_config.json.


PoolBase.registerMBeans fails when multiple wars in same container

If I have two wars deployed to the same Tomcat container, each war with one or more connection pools, PoolBase.registerMBeans fails when the pool names happen to coincide.  This happens if I set the poolName myself or if I rely on the Hikari generated pool names.  See example below.
HikariPool-0 - You cannot use the same pool name for separate pool instances.
HikariPool-0 - You cannot use the same pool name for separate pool instances.

Thanks Brett!  I'll give it a shot.

Brett, this is what I ended up doing:
    <bean id="analyticsDataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">
        <constructor-arg index="0">
            <bean class="com.zaxxer.hikari.HikariConfig">
                <property name="registerMbeans" value="true"/>
                <property name="poolName">
                    <bean class="com.digitalglobe.util.PoolNumberPrepender" factory-method="prependPoolNumber">
                        <constructor-arg index="0" type="java.lang.String" value="${services.common.analytics.db.poolName}"/>
                    </bean>
                </property>
                <property name="validationTimeout" value="${services.common.analytics.db.validationTimeout}"/>
                <property name="leakDetectionThreshold" value="${services.common.analytics.db.leakDetectionThreshold}"/>
                <property name="maxLifetime" value="${services.common.analytics.db.maxLifetime}"/>
                <property name="idleTimeout" value="${services.common.analytics.db.idleTimeout}"/>
                <property name="connectionTimeout" value="${services.common.analytics.db.connectionTimeout}"/>
                <property name="minimumIdle" value="${services.common.analytics.db.mininumPoolSize}"/>
                <property name="maximumPoolSize" value="${services.common.analytics.db.maximumPoolSize}"/>
                <property name="driverClassName" value="${services.common.analytics.db.driver}"/>
                <property name="jdbcUrl" value="${services.common.analytics.db.jdbcUrl}"/>
                <property name="username" value="${services.common.analytics.db.user}"/>
                <property name="password">
                    <bean factory-bean="decryptor" factory-method="decryptString">
                        <constructor-arg value="${services.common.analytics.db.password}"/>
                        <constructor-arg>
                            <bean factory-bean="decryptor" factory-method="getSecretKeyFromFile"/>
                        </constructor-arg>
                    </bean>
                </property>
            </bean>
        </constructor-arg>
    </bean>

And then in PoolNumberPrepender, I do something similar to what you're doing with the generic pool names.  You should be able to do something similar with the provided pool names and then I wouldn't have to do the above.  Keep me posted.
Thansk in advance!

I am going to mark this as closed.  While it theoretically possible for HikariCP to try to guarantee pool name uniqueness across classloader containers with user-specified pool names, this seems like something better left to the user of the pool, as @victropolis has done above.


better no value found errors

Feature request for Node 3.0.
It would be very nice if node errors included their keys. This will probably take some work to implement since we will need to pass the keys around to different node objects as we recurse. We can probably look into how Encodable/Decodable do it with the keyed container API.
Example of current non-helpful errors:
[Node Error: No value found at path '', expected 'String'] [Identifier: Node.NodeError.unableToConvert] [Possible Causes: typo in key path, underlying type is not convertible, unexpected '.' being interpreted as path instead of key] [Suggested Fixes: called `get(...)` on a key or key path that does not exist in the data, the data being parsed is missing required values or is incorrectly formatted, found unconvertible data, e.g., got a string of letters when an integer is required, if you have keys containing a '.' that shouldn't be interpreted as a path, use 'DotKey("actual.key")']

For code:
hosting.gitURL = try req.data.get("gitURL")

This should say something like "no value found at path gitURL".  Would be even better if it suggested some possible fixes (for example there was only one other key in the dictionary, named "gitUrl", etc).

@tanner0101
Same issue. Any solutions?

What do you mean? This repository has been dead for more than a year.


Issue: Performance/Memory Challenges in Large Domains

Issue: When running the tools against a domain with over 100k users, or large numbers of OUs and ACLs, script consumes upwards of 4GB or more of RAM, ~20% proc, and generally consumes all Disk IOPs which, when run from a laptop with 8GB of RAM, brought it to an unusable level. In the event the script consumes too much overhead, the system crashes, resulting in having to start completely over. On older domains (2008 R2), large queries can result in timeouts in the domain web services causing 'invalid enumeration context' errors.
Runtime Environment:
Windows 10 v1803
PowerShell v5.1
Module Versions - Latest of all dependent modules, pulled directly from GitHub
Expectations:
a) When possible, data for each area should be written out to files as it's pulled, and then cleared from memory, to minimize memory footprint.
b) Long running tasks, such as collecting OU ACLs, should have a progress bar to provide an indication of elapsed time and percent complete.
c) By making use of frequent data dumps to the file system, it should be possible to enable a resume in the event that a given run crashes the system, or some other factor causes the process to be interrupted.
Suggestion: It might be beneficial to pull a count of objects and performing some pre-analysis on the number of objects prior to initiation of a section run. For larger environments, it should be possible to break processing up into smaller chunks, perhaps using filters (ie found 100k objects, find all users with given name starting with a, b, or c, then do d,e,f, etc until all objects have been processed). This information can be dumped to disk, and then the files processed individually into the final formats at the end of the run. It may also be beneficial to switch from using the AD cmdlets to using the legacy ADSI interface in larger environments, since it doesn't have the same timeout limitations.

I've encountered similar issues on even smaller scale domains.
So indeed there is a big problem with that one, and it needs to be addressed. I'm even wondering if building a large final object will require saving to disk every section as well.
One issue that you briefly mentioned is exporting all properties, and that is something that needs to be limited. But since I didn't know what people may want I didn't want to add it just yet. But recently I've been working on PSWinReporting where you get to define anything you want, in a language you want and was thinking to bring this approach into PSWinDocumentation:
https://github.com/EvotecIT/PSWinReporting/blob/9b351e5c7bf523704d4f5c458c22ef5b0155613c/Private/Parameters/Script.ReportDefinitions.ps1#L404-L570
The way it works is that on the left-hand side in Fields section you define fields that you want each type of events to get and the right side will be the one it will be displayed over. In the case of PSWinDocumentation, each section could have similar Fields section where each Type would have its definition. The biggest issue with such an approach is that some objects are related to each other. For example, I didn't want to ask AD multiple times for User's manager so I just ask once and then build Manager data from Users. My problem is that if people will start removing fields errors would occur.
I guess I could make 2 Fields sections, one that is static, required, and other which is optional that people can remove.
My main problem with any of my tools is design. I sometimes spend days building script and then rebuild config file multiple times because I can't get it right. Feel free to make suggestions or even submit Pull Requests :-) I'm happy to get help as clearly there are lots of issues and lots of ground to cover. If you can lay out how config file should, could look like in your opinion that would be great.
BTW. I do plan export to Dynamic HTML as well at some point but again... performance issue is a priority, that may even require a complete rewrite to some extent.

Thanks so much for getting back to me. I was actually intending to go back and update one of my logged items, as I was able to chase down the source of at least part of the issue, though I encountered another as the Word document didn't generate, even though the Excel file did.
In this particular domain I am pulling information for, not only is it fairly large (more than 180k user objects), but it is a 2008 R2 domain and the company has deployed a number of custom attributes to AD. I was able to retrieve data for all of the other domains without error, but the big one kept breaking for groups, computers, and users, though it got all the other data fine. The issue ended up being two-fold;

The large size of the environment, when paired with retrieving all attributes, results in a query that takes longer than the allowed 30 minute timeout on the AD web service that the AD cmdlets uses. You can increase the timeout, but this is a security risk and not a recommended practice.
 - The second issue was the manner in which the custom attributes were deployed, which allowed them to be tied to user, computer, and group objects, even though they currently only use the attributes for users. That itself isn't necessarily a problem, but they also didn't set the attributes as indexed, so this slows the request down even further and results in an enumeration error. The error isn't currently trapped by your module, so the process proceeds as if everything is fine.
Currently, even if I turned off everything for say, group objects, the script still retrieves all of the objects at the beginning of the run, which takes up memory unnecessarily, particularly when retrieving all properties, though I understand the challenge around meeting user expectations. I have several ideas on how you could make some possibly minor enhancements to improve efficiency, without having to resort to using ADSI.

Adjust the logic to skip retrieval of an given object type entirely if all items and dependencies are turned off in the config for that type. This will save a lot of overhead if someone doesn't want a particular item.

Instead of retrieving all properties for all objects of each type up front and holding it in memory, which hits issues when targeting large domains or a large number of objects with non-indexed properties, retrieve the objects with the default properties only to keep in memory.

Once you get to the section where you actively process a given object type, you can make several adjustments

Adjust the script to allow for a default of all properties, or for the person running to provide a string collection of specific properties

To optimize the default of all properties, but still exclude key properties, create a helper function (that fires only for the default) to retrieve a single AD object of that type, run it through Get-Member to get the property names, pass it through a select (like you do now) to filter out the problem attributes, then return a string collection of the remaining properties (since the MS AD cmdlets don't accept anything other than strings)

Once you have your property name collection (as a string), do a foreach($object in $objects){} loop to retrieve each object with all desired properties one at a time

As each object is returned, dump it to a file instead of storing it in memory...possibly to CSV

Use one of the several modules available that allow interacting with a CSV as if it were a table resource to selectively query items for the various reports, like admin users, password never expires, etc
By doing things like this, you get a lot of benefits, like avoiding errors due to lack of indexing, the ability to provide a progress indicator using something like Write-Progress (you can do some nifty stuff with parent and child progress windows), as well as my personal favorite in this instance, which is the ability to resume without having to start from zero if something causes an interruption. The obvious trade-off is a bit more time to run overall, but with feedback to the console, the reduced memory footprint, and the ability to resume, I think most people would be fine with it.
As far as my thoughts on the config, I'm just thinking of a more hierarchical approach when it comes to the objects, like you did with the Forest and Domain pieces:
SectionDomain = [ordered] @{    SectionDomainOverview = [ordered]@{
        Enable = $true
        DomainSubSections = [ordered]@{
            DomainIntroduction = $true
            DomainControllers = $true
            DomainFSMO = $true
            DomainTrusts = $true
            DomainDNS = [ordered]@{ (include some DNS zone info here perhaps, SOA, which servers are running DNS, self-pointed, etc)
                Enabled = $true                DomainDNSSrv = $true
                DomainDNSA = $true
            }
        }
        etc
    }
    SectionDomainGroupPolicies = [ordered]@{
        Enable = $true        GroupPoliciesSubSections = [ordered]@{            GroupPoliciesDetails = $true
            GroupPoliciesACL = $true        }
    }
    SectionDomainOrganizationalUnits = [ordered]@{
        SectionDomainOrganizationalUnitsBasicACL
        SecitonDomainOrganizationalUnitsExtendedACL
    }    SectionDomainUsers = [ordered]@{
        Enable = $true
        InScopeProperties = * (allow to specify a comma separated string 'item1,item3,item54' in place of *)
        DomainUsersSubSections = [ordered]@{            SystemAccounts = $true
            NeverExpiring = $true
            NeverExpringIncDisabled = $true
            etc
        }
    }
}
You could obviously make each of the subsections definitions their own ordered hash tables if you wanted to specify additional settings. The settings for things like Excel worksheet names, document heading or table types and the like, I would pull up into their own main section and variablize everything. I can specify a table look for every table item, and every header style to be applied, and they would relate directly to the hierarchy of the other sections. This reduces the code required and ensures consistency between document sections without a lot of effort on the part of the person running the script. The hierarchy also clearly shows where the dependencies are...can't get system account information if I didn't enable getting users for example. You can also add in comments to show things like 'this section outputs to Excel only' or 'selecting this option will add X property to any custom property specification' etc.
As to doing a pull request, I would actually love to, once I have some time to spend on actually doing the work, but right now the best I can do is provide some feedback. I came across your module while trying to quickly pull together content for a customer and wanting to save some time converting data to documents. Now, of course, it's the holidays and I'm going on PTO for the next two weeks.
I've actually been working on a similar project to this one on my own that is currently at the concept phase, but looking at a different data collation approach, possibly using SQLite or some similar flat file database to dump data into. Not only will this provide at least some limited protection of the collected data, but then I can more readily separate collection from document generation for better modularity, as well as tie it in to things like Power BI or Visio for added visualizations to go with the docs. I am making some headway on this piece presently, but only in relation to another pet project that will use a similar data approach, though the groundwork for the one effort will feed right into the discovery/documentation project. Perhaps we can work on a collaboration that combines our two efforts once I get this other project knocked out?
Chris Whitfield

On Wednesday, December 19, 2018, 5:11:29 PM CST, Przemysław Kłys<EMAIL_ADDRESS>wrote:

I've encountered similar issues on even smaller scale domains.
So indeed there is a big problem with that one, and it needs to be addressed. I'm even wondering if building a large final object will require saving to disk every section as well.
One issue that you briefly mentioned is exporting all properties, and that is something that needs to be limited. But since I didn't know what people may want I didn't want to add it just yet. But recently I've been working on PSWinReporting where you get to define anything you want, in a language you want and was thinking to bring this approach into PSWinDocumentation:
https://github.com/EvotecIT/PSWinReporting/blob/9b351e5c7bf523704d4f5c458c22ef5b0155613c/Private/Parameters/Script.ReportDefinitions.ps1#L404-L570
The way it works is that on the left-hand side in Fields section you define fields that you want each type of events to get and the right side will be the one it will be displayed over. In the case of PSWinDocumentation, each section could have similar Fields section where each Type would have its definition. The biggest issue with such an approach is that some objects are related to each other. For example, I didn't want to ask AD multiple times for User's manager so I just ask once and then build Manager data from Users. My problem is that if people will start removing fields errors would occur.
I guess I could make 2 Fields sections, one that is static, required, and other which is optional that people can remove.
My main problem with any of my tools is design. I sometimes spend days building script and then rebuild config file multiple times because I can't get it right. Feel free to make suggestions or even submit Pull Requests :-) I'm happy to get help as clearly there are lots of issues and lots of ground to cover. If you can lay out how config file should, could look like in your opinion that would be great.
BTW. I do plan export to Dynamic HTML as well at some point but again... performance issue is a priority, that may even require a complete rewrite to some extent.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks for your feedback. I've some ideas on how to do things but it will take time. Feel free to jump in when you're comfortable enough.
Btw. you do know this module is able to export to SQL (MS SQL) right? Just like it does for Excel, it does the same magic with SQL so you can build your reporting based on that. I will add support to HTML (already working on PSWriteHTML module) which should help people use things as they want. There are few people having similar projects to mine like Brad Wyatt and if we join forces try to fix this one correctly it will be easier :-) 3 - 4 people working on one module, making sure it works from 10 to 200k users would be great.

Yes, I saw that your module does export to SQL, but from what I saw in the code, it's based on a full SQL Server installation, and it's a dump of data at the end of the process. While my approach does end with data in a SQL-esque location, there are some marked differences.

it is not full SQL, but SQLLite, which essentially consists of just a DLL and a flat file, but with many of the optimizations and speed you would get from one of the bigger products, making it highly portable while also eliminating challenges around setup and security, since nothing other than the module itself, and possibly the the sub-module for interaction with the DB instance, need to be 'installed' anywhere.
I am intending to go a step further and eliminate all flat file storage during runtime, to include the advanced config stuff for other output options. For example, you have a great idea with providing a whole framework for turning elements on or off, and setting formatting in your output and the like. What I'm thinking of is a table in the database file that contains all my preferences, and a set of functions to retrieve and update those settings. There is even a SQLite module that will mount the DB via a provider, which should reduce the T-SQL required to write, retrieve, and update data.
I'm working through an approach that will essentially allow me to more or less stream the data into the database file directly, and therefore enabling me to substantially reduce the memory footprint and giving me a full relational data source to work with more or less offline
If I use full SQL, even if I use Express, that has to be downloaded and installed somewhere, and then I have to build the tables (though I did see a module that will dynamically build the table based on the object in the pipeline), and I won't be able to persistently keep my preferences within that more flexible format. Yes, I could simply create a setup utility that creates the table and loads in my defaults, but then I have to put all that into code, so it just seemed more efficient to use something in between.

I fully agree that having a handful of dedicated people all working on a single module is more efficient than multiple separate approaches and then trying to combine. My challenge at the moment is that I need some of these underlying items for a customer facing effort rather immediately, and I'm taking some shortcuts to get to the end result. For example, while trying to figure out the source of the problem I was having with this module, I ended up firing elements of your module code manually, but I replaced the ActiveDirectory module commands with the Quest ActiveRoles module, which doesn't use the AD Web service, and uses pure .NET under the covers. It's getting me what I need immediately, but it's not something you could redistribute, and it doesn't solve the other refactoring issues.
I'm also simultaneously working on a reporting type effort that involves Jira and Power BI, and I'm using PowerShell to retrieve the data and write to a file that I then consume with Power BI for reporting. Problem is that the customer now wants historical and change data as well, and that was going to get ugly to manage with flat files, but is easy in a database, which led to the portability bit, and now I'm here. In a couple months, these immediate needs should ease, and I can work with you guys on your project. In the meantime, just trying to contribute what I can, which is suggestions.
Btw, along those lines, if you are interesting, I have put together some pieces from around the web to pull some schema details...namely a schema update history report and some determination on which schema extensions are in place for Microsoft products, updated with the latest versions. I'd be happy to share the snippets if you think it would be useful in your documentation module.
Chris Whitfield
On Friday, December 21, 2018, 1:33:08 PM CST, Przemysław Kłys<EMAIL_ADDRESS>wrote:

Thanks for your feedback. I've some ideas on how to do things but it will take time. Feel free to jump in when you're comfortable enough.
Btw. you do know this module is able to export to SQL (MS SQL) right? Just like it does for Excel, it does the same magic with SQL so you can build your reporting based on that. I will add support to HTML (already working on PSWriteHTML module) which should help people use things as they want. There are few people having similar projects to mine like Brad Wyatt and if we join forces try to fix this one correctly it will be easier :-) 3 - 4 people working on one module, making sure it works from 10 to 200k users would be great.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Feel free to provide whatever you can. This module is not just for me but for everyone. I already have some more stuff in my mind what data is needed so if you can provide code snippets as part of an issue feel free to do so. If I will get time I will incorporate it. If not.. you will do it yourself later on.
Generally, the goal for this tool is to have fully functional documentation & audit module in one. I want to go to the Client and be ready for anything. And I need it to just work.
So when you will have time to drop your ideas, drop problems, drop issues, and drop PR's when you have time. I willl rebuild part of module to make sure it's a bit modular and easier to work with.